PaperlayerView exact revision on arXiv

Source-linked AI summary

APRES: An Agentic Paper Revision and Evaluation System

Bingchen Zhao, Jenny Zhang, Chenxi Whitehouse, Minqi Jiang, Michael Shvartsman, Abhishek Charnalia, Despoina Magka, Tatiana Shavrina, Derek Dunfield, Oisin Mac Aodha, Yoram Bachrach

arXiv:2603.03142v1cs.CLcs.AI

TL;DR

Peer review faces strain and inconsistent evaluation, limiting authors’ feedback on improving manuscripts. APRES uses LLMs to discover impact-predictive criteria and guide automated revisions, improving predicted citation counts and producing papers preferred by human evaluators.

  • Problem

    Peer review faces strain and inconsistent evaluation, making it difficult for authors to receive feedback that improves their scientific communication.

  • Method

    APRES uses LLM agents to discover criteria predictive of future impact and guide a closed-loop process for revising papers’ presentation.

  • Results

    APRES successfully revises papers to improve their predicted citation counts, as supported by automated and human evaluations.

  • Takeaways & Limitations

    APRES supports using LLMs to help authors stress-test manuscripts and improve clarity while augmenting rather than replacing human experts.

  • Takeaways & Limitations

    APRES analyzes only paper text, does not process figures, and cannot perfectly guarantee that revisions preserve scientific content.

Abstract

from arXiv · show

Scientific discoveries must be communicated clearly to realize their full potential. Without effective communication, even the most groundbreaking findings risk being overlooked or misunderstood. The primary way scientists communicate their work and receive feedback from the community is through peer review. However, the current system often provides inconsistent feedback between reviewers, ultimately hindering the improvement of a manuscript and limiting its potential impact. In this paper, we introduce a novel method APRES powered by Large Language Models (LLMs) to update a scientific papers text based on an evaluation rubric. Our automated method discovers a rubric that is highly predictive of future citation counts, and integrate it with APRES in an automated system that revises papers to enhance their quality and impact. Crucially, this objective should be met without altering the core scientific content. We demonstrate the success of APRES, which improves future citation prediction by 19.6% in mean averaged error over the next best baseline, and show that our paper revision process yields papers that are preferred over the originals by human expert evaluators 79% of the time. Our findings provide strong empirical support for using LLMs as a tool to help authors stress-test their manuscripts before submission. Ultimately, our work seeks to augment, not replace, the essential role of human expert reviewers, for it should be humans who discern which discoveries truly matter, guiding science toward advancing knowledge and enriching lives.

1 Introduction

APRES is an agentic LLM framework that discovers evaluation criteria predictive of future paper impact and uses them to guide selective, automated manuscript revision. It aims to make peer-review support more reliable while preserving the paper’s core scientific content and augmenting human expertise.

  • Motivation: Peer review faces unprecedented strain as submission volumes outpace the growth of qualified reviewers, causing reviewer fatigue and lengthy review cycles.These pressures hinder consistent feedback and manuscript improvement.
  • Method: APRES uses LLMs to discover evaluation criteria predictive of a paper’s future citation impact before revising its text.The framework treats rubric discovery as distinct from revision guidance.
  • Method: The framework’s two stages share an agentic search scaffold: rubric search identifies predictive criteria, then paper improvement iteratively revises the manuscript using that rubric.The discovered rubric guides revisions intended to maximize predicted impact while selectively changing the paper’s text.
  • Validation: APRES is validated through rubric-search and automated-revision experiments that test impact prediction and meaningful manuscript improvement.The appendix also evaluates the reliability of the LLM evaluation pipeline by replicating NeurIPS consistency-study designs.
  • Implications: The work positions agentic LLMs as tools for more reliable, data-informed review support that augment rather than replace human reviewers.This motivation aligns with emerging conference programs using AI-generated reviews and LLM feedback to improve human review.

2 Related Work

Related work spans LLM-assisted review generation, peer-review reliability, scholarly impact prediction, and automated scientific-text revision. APRES combines these directions by integrating predictive evaluation criteria with closed-loop paper revision.

  • LLMs for review generation and author assistance: LLM review research has progressed from template-based comments to structured pipelines, multi-agent deliberation, tree-of-thought methods, and reinforcement-learned reviewers.Large-scale studies find LLM feedback overlaps substantially with human reviews, though controlled experiments report modest utility in real workflows.
  • LLMs for review generation and author assistance: 70% of surveyed authors revised their submissions after using LLM-based checklist assistants in the NeurIPS 2024 pilot study.These assistants help authors align manuscripts with venue guidelines.
  • Reliability of peer review: Peer-review reliability studies report that independent committees disagreed on roughly a quarter of papers in NeurIPS 2014 and 2021.This work connects APRES to longstanding concerns about arbitrariness in review decisions.
  • Impact prediction and scholarly recommendation: Impact-prediction methods have evolved from metadata and citation graphs to document embeddings and text-based LLM forecasting of normalized citations.Readability and accessible writing styles have also been associated with greater scientific impact, understanding, and reader confidence.
  • Automated text revision for scientific writing: Automated scientific-writing revision has advanced from grammatical correction and transformer-based summarization to multi-agent systems that provide constructive revision feedback.APRES is presented as the first method to integrate predictive evaluation-criteria discovery with a closed-loop automated paper-revision system.

3 Method

APRES uses agentic LLM search to discover a citation-predictive review rubric, then applies that rubric in an iterative scaffold for revising papers toward greater quality, clarity, and readability. The revision process is instructed to preserve experimental results while changing presentation.

  • Predicting Citations via Agentic Search: APRES defines paper impact through citation count and searches for rubric items whose evaluation scores predict citations twelve months after publication.Citation counts are modeled as non-negative, often overdispersed data using negative binomial regression.
  • Predicting Citations via Agentic Search: A Rubric Proposer iteratively proposes or refines k review items, while a Reviewer scores each paper on those items to create feature vectors.Rubric items assess aspects such as originality, clarity of presentation, and technical soundness.
  • Predicting Citations via Agentic Search: A negative binomial model predicts citation counts from reviewer features using MAE, and the search selects and refines the best rubric to improve predictive accuracy.This closed loop searches for evaluation criteria aligned with future scientific impact.
  • Iterative Paper Revision: The revision scaffold uses the discovered rubric R* as a surrogate objective, generating feedback, rewriting the paper, re-evaluating it, and iterating on the highest-scoring version.The process operationalizes paper improvement despite future impact being unknown at writing time.
  • Iterative Paper Revision: The Rewriter is instructed not to alter experimental results, restricting revisions to presentation while maximizing the discovered rubric score.This allows the system to generate revised papers from an original paper and rubric without changing experimental findings.

4 Experiments

Experiments use a large contemporary machine-learning paper and review dataset to evaluate citation-impact prediction and LLM-guided paper revision. MultiAIDE discovers a low-error impact rubric, while APRES improves predicted impact across paper categories, with gains constrained when underlying method performance cannot be changed.

  • Dataset: The study uses papers and peer reviews from ICLR 2024–2025 and NeurIPS 2023–2024, totaling 26,707 papers with influential-citation data from Semantic Scholar.Influential citations are intended to capture deeper scientific engagement rather than passing mentions.
  • Citation Number Prediction: MultiAIDE achieves the lowest citation-prediction MAE across tested methods, converging rapidly before exploring the solution space.The search reaches below 2.0 MAE for Gemini 2.5 Pro and outperforms the tested baselines, including Prompt breeder.
  • Citation Number Prediction: 1.92 MAE is achieved by o3, compared with 2.25 for o1, 1.96 for Gemini 2.5 Pro, and 2.30 for Gemini 2.5 Flash.These are the reported cross-model MAE scores for MultiAIDE search.
  • Paper Improvement: The Rewriter increases predicted impact scores for every paper category, showing that LLM-generated feedback can produce concrete textual improvements.Revision uses diff-based search/replace edits, and improvement is measured as ΔS = Srev − Sori.
  • Paper Improvement: Revision cannot improve the underlying method’s performance because it changes presentation and contribution description without rerunning experiments, limiting gains for performance-based rejections.Papers rejected for presentation-related reasons may benefit more from revision.
  • Paper Improvement: 3.33 ΔS for o3 and 2.86 ΔS for Gemini 2.5 Pro are reported for borderline papers, while clear-accept papers show smaller gains consistent with quality saturation.Clear-reject papers improve but plateau at a substantially lower quality ceiling; their reported ΔS values are 2.98 for o3 and 3.34 for Gemini 2.5 Pro.

5 Discussion

The discussion argues that LLMs may contribute most by discovering effective evaluation criteria rather than mimicking human reviewers, while paper revision still needs deeper technical feedback and human involvement. It also identifies limitations involving citation-based impact measurement, omitted figures, and possible inadvertent inaccuracies.

  • From Mimicking Humans to Predicting Impact: LLMs may be more valuable for discovering novel, effective evaluation criteria than for mimicking human reviewers.APRES’s success compared with human-like review guidelines motivates this interpretation.
  • From Mimicking Humans to Predicting Impact: Citation counts are treated as a proxy for paper impact, but this measure is subject to multiple biases.The passage notes that citation numbers may reflect factors such as the number of researchers working on a topic.
  • The Path to More Helpful Reviews: APRES improves papers, but its focus on surface-level writing suggests a need for deeper, more constructive technical feedback.The discussion connects this need to the challenges authors experience with iterative technical feedback.
  • The Path to More Helpful Reviews: LLM-based revision systems should consider more human-in-the-loop interactions because deep technical feedback can increase author frustration and reduce submissions.This recommendation is based on analysis by Goldberg et al. (2024).
  • Limitations: APRES analyzes only paper text, does not process figures, and cannot perfectly guarantee that revisions preserve scientific content.The LLM could inadvertently introduce subtle inaccuracies while modifying presentation.

6 Conclusion

APRES automatically improves scientific papers’ clarity and readability using a rubric discovered through a proxy for scientific impact. Its iterative revisions improve predicted citation counts, supporting LLMs as manuscript stress-testing tools that augment rather than replace human experts.

  • 6 Conclusion: APRES automatically improves the clarity and readability of scientific papers using a rubric optimized for a proxy of scientific impact.The approach is designed to improve text without advocating replacement of human expertise.
  • 6 Conclusion: APRES’s iterative revisions improve papers’ predicted citation counts, as verified through automated and human evaluation.The conclusion reports agreement between both evaluation modes without specifying their numerical results here.
  • 6 Conclusion: The findings support using APRES to help authors stress-test manuscripts and improve textual clarity while augmenting, not replacing, human experts.The system is presented as an assistive tool for manuscript improvement rather than a substitute for expert judgment.

Ethics Statement

APRES is intended to augment expert peer review by improving presentation without changing scientific content, while acknowledging risks from citation-based objectives and AI-assisted revision. The authors address these risks through edit constraints, disclosure guidance, human oversight, public-data safeguards, and transparency practices.

  • Purpose and oversight: APRES supports authors as a pre-submission assistant and is intended to augment, not replace, expert peer review.The authors align their practices with the ICLR Code of Ethics and norms concerning research integrity, fairness, transparency, and harm avoidance.
  • Data and human subjects: The study uses public manuscripts, reviews, and citation counts accessed under venue or author terms, without private-material scraping, deanonymization, or access-control circumvention.Human evaluation used consenting, compensated adult annotators in a blind pairwise-preference setup, collected no personally identifiable information, and involved minimal risk without intervention.
  • Risk of harm and dual use: APRES restricts revisions to organization, clarity, and exposition, explicitly prohibiting changes to technical claims, reported results, or tables.The authors identify potential misuse including gaming perceived impact, influencing reviewers, homogenizing writing, and disadvantaging out-of-distribution communities.
  • Risk of harm and dual use: The discovered rubric is treated as a surrogate objective rather than a definitive quality measure, with human experts remaining the ultimate arbiters of scientific merit.The authors recommend disclosure when AI assistance materially affects a manuscript and describe misuse monitoring as part of their precautions.
  • Fairness and transparency: Citation-based rubrics may inherit field-, venue-, trend-, language-, and popularity-related biases, motivating strata analyses, failure-case studies, distribution-shift analysis, and alternative objectives.The authors do not filter by or infer author identity or demographics, and plan to share code, prompts, derived non-identifying labels or metrics, and reproducibility documentation.

Reproducibility Statement · Appendix · A Effectiveness of LLM reviewers

The paper documents APRES implementation and preprocessing details using public datasets and plans to release replication materials. An appendix study evaluates LLM review systems against conference outcomes and human-reviewer consistency.

  • Reproducibility Statement: APRES implementation details are documented to support reproducibility.The documentation covers the system’s implementation in the main paper and appendix.
  • Reproducibility Statement: Model prompts, search hyperparameters, and evaluation pipelines are documented in the paper and appendix.
  • Appendix: The datasets come from publicly available OpenReview submissions and Semantic Scholar citation counts.The appendix describes the preprocessing steps applied to these datasets.
  • Appendix: The authors will release the full codebase, instructions, derived data splits, and evaluation scripts upon publication.
  • A Effectiveness of LLM reviewers: The additional study examines how well LLM reviews correlate with actual conference outcomes.
  • A Effectiveness of LLM reviewers: The study also examines consistency between LLM review systems and human reviewers.

A.1 Glicko2 rating

APRES evaluates papers with an LLM-judged, pairwise Glicko2 rating system that produces a continuous quality ranking. The ratings show a clear, strong positive correlation with official conference decisions, with higher-rated papers increasingly receiving accepted-paper outcomes.

  • Method: APRES generates rubric-based reviews, samples paper pairs, and asks an LLM judge to compare them for pairwise Glicko2 rating.The system uses the discovered APRES rubric to generate reviews before comparing randomly sampled paper pairs.
  • Results and Analysis: The Glicko2 system creates a continuous quality ranking for all papers and validates it against official conference outcomes.The ranking is evaluated by comparing LLM-derived ratings with final conference decisions.
  • Method: The comparison prompt provides both papers and their detailed reviews, asking which is better overall across six evaluation dimensions.The dimensions are technical soundness and correctness, novelty and originality, significance and impact, clarity, experiments and evaluation, and overall contribution.
  • Method: The judge records confidence, reasoning, a 1-10 score difference, and the winning paper for each comparison.The required output includes confidence on a 1-5 scale and identifies whether paper A or B wins.
  • Results and Analysis: As Glicko2 ratings increase, rejected papers become less prevalent while posters, spotlights, and orals steadily increase.Figure A1 visualizes this relationship between LLM-derived ratings and conference outcomes.

A.2 Reviewer Consistency

APRES evaluates reviewer consistency by converting Glicko2 quality ratings into binary accept/reject decisions and comparing disagreement across LLM committees and human benchmarks. The results indicate that LLM-based evaluation provides a more consistent and stable signal of paper quality than human peer review.

  • Methodology: LLM evaluation assigns ‘Accept’ to papers in the top 25th Glicko2-rating percentile and ‘Reject’ to the remaining 75%.The indirect thresholding approach avoids directly prompting accept/reject classifications and reduces the burden of global absolute-quality judgments.
  • Methodology: Consistency is measured by pairwise Disagreement Rate across complete binary-decision sets for distinct models, repeated runs, and human benchmarks.The comparison matrix includes human disagreement benchmarks of 23% for NeurIPS 2021 and 25.9% for NeurIPS 2014.
  • Results and Analysis: Figure A2 shows that LLM-based evaluation is significantly more consistent than human peer review, especially within the same model family.The figure compares OpenAI-o1, OpenAI-o3, Gemini 2.5 Pro, and Gemini 2.5 Flash committee configurations.
  • Results and Analysis: The consistency results support LLMs as a stable, less random signal of paper quality and validate APRES for producing meaningful research-paper revisions.The conclusion follows from the observed reliability of the LLM-driven evaluation pipeline relative to the stochasticity of human review.

C Dataset Information · D Paper Improvement Using Other LLMs

The dataset comprises papers downloaded from OpenReview, parsed into editable text while excluding author information, and characterized by citation distributions and conference-based review categories. Additional experiments with o1 and Gemini 2.5 Flash show revision-improvement trends consistent with the main paper.

  • C Dataset Information: ICLR papers were classified as Clear Accept above 6, Clear Reject below 4, and Borderline otherwise by average human-review scores.The ICLR borderline review score is 5.
  • C Dataset Information: NeurIPS papers were classified as Clear Accept above 5, Clear Reject below 3, and Borderline otherwise by average human-review scores.The NeurIPS borderline review score is 4.
  • C Dataset Information: Papers were downloaded as PDFs from OpenReview and processed with the SciPDF parser.The parser supplied the textual input used for subsequent APRES processing.
  • C Dataset Information: SciPDF separates main text, figures, and tables, enabling programmatic control over which sections APRES edits.Parsed author information was excluded to prevent information leakage.
  • C Dataset Information: Figure A3 shows the distribution of citation counts for papers in the dataset.The dataset’s citation histogram is referenced for further information in Section 4.
  • D Paper Improvement Using Other LLMs: Experiments using o1 and Gemini 2.5 Flash produced paper-improvement results in Figures A4 and A5.These experiments evaluated revision improvement with alternative language models.
  • D Paper Improvement Using Other LLMs: The general trend in revision improvement with o1 and Gemini 2.5 Flash was consistent with the main paper’s results.The comparison is made against the trend shown in Figure 4.

E Discovered Rubric · F Prompts

APRES discovers a paper-reviewing rubric with over 60 items spanning problem significance, literature context, methodology, results, discussion, novelty, writing quality, and future impact. The rubric also specifies detailed criteria and 0/5/10 anchors for evaluating these dimensions.

  • E Discovered Rubric: The discovered rubric contains over 60 items covering eight dimensions of scientific-paper quality.These include problem formulation and significance, literature review and context, methodology and technical rigor, results and analysis, discussion and conclusion, originality and novelty, writing and presentation quality, and future impact and influence.
  • E Discovered Rubric: Problem criteria assess clarity, field relevance, timeliness, scope, motivation, hypothesis clarity, and potential impact.Several criteria use 0, 5, and 10 anchors ranging from unclear or irrelevant problems to clear, timely, well-scoped, and strongly motivated contributions.
  • E Discovered Rubric: Literature criteria evaluate comprehensiveness, accuracy, gap identification, critical analysis, reference recency, and positioning against alternatives.The anchors distinguish missing or outdated literature and superficial comparisons from comprehensive, accurate, current, critically synthesized reviews with insightful comparisons.
  • E Discovered Rubric: Methodology criteria cover appropriateness, replicability, justification of choices, technical correctness, assumptions, data quality, baselines, ethics, and limitations.The rubric emphasizes sufficiently detailed methods, technically correct implementation, strong comparisons, explicit assumptions, appropriate data, ethical discussion, and candid acknowledgment of limitations.
  • E Discovered Rubric: Results criteria assess presentation clarity, support for claims, analytical depth, statistical validity, robustness, negative-result reporting, interpretation soundness, and significance.Criteria distinguish confusing or weakly supported results from clear presentations, convincing evidence, deep analysis, rigorous statistics, transparent reporting, and cautious interpretation.
  • E Discovered Rubric: Discussion criteria evaluate synthesis, contribution articulation, implications, generalizability, limitations, and future-work suggestions.The anchors favor discussions that connect findings to research questions, clearly state contributions, examine broader implications and boundaries, acknowledge limitations, and propose specific future work.
  • E Discovered Rubric: Novelty, writing, and impact criteria assess conceptual, methodological, empirical, and synthetic originality; organization and language; and prospects for insight, citation, adoption, scalability, and elegance.The rubric includes whether work advances the state of the art, communicates clearly and concisely, changes how a problem is viewed, supports future work, and reaches other fields.

G User Study Annotation Guidelines · H Example Papers

The user study compares human- or LLM-generated paper excerpts using NeurIPS-inspired criteria focused on presentation while holding scientific contributions constant. Supplementary materials also provide before-and-after APRES paper examples in the accompanying zip file.

  • G User Study Annotation Guidelines: Annotators compare two paper-excerpt versions, select a preference, and briefly justify their choice using specified evaluation criteria.Excerpts may include abstracts, introductions, or specific sections, and may be human- or LLM-generated.
  • G User Study Annotation Guidelines: The evaluation follows NeurIPS reviewing principles and emphasizes writing and presentation because both versions retain the same scientific contribution.The guidelines explicitly guarantee that the papers’ contributions are the same.
  • G User Study Annotation Guidelines: Reviewers assess readability, precision, logical flow, message clarity, technical correctness, contribution framing, grammar, tone, and concision.These criteria cover both effective communication and preservation of important technical details.
  • G User Study Annotation Guidelines: A neutral judgment is acceptable when both versions are considered perfectly acceptable, but neutral responses should be used very sparingly.The guidelines request a preference or, only when warranted, no strong preference.
  • G User Study Annotation Guidelines: The examples prefer Version B when its direct, concise wording makes the contribution easier to grasp without unnecessary jargon.Version B replaces a long sentence with shorter statements about the proposed attention mechanism and benchmark performance.
  • G User Study Annotation Guidelines: The examples prefer Version A when simplification omits precise technical information, such as non-linear degradation on highly skewed distributions.The justification treats those omitted details as critical to correctness and real-world interpretation.
  • H Example Papers: Before-and-after APRES revision examples are provided in the same zip file as the supplementary material.The examples illustrate papers before and after APRES revision.
Loading 2603.03142v1…