Source-linked AI summary
ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews
Mike D'Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, Doug Downey
TL;DR
Scientific-paper revision from peer feedback requires interpreting complex comments, yet datasets for studying these tasks are limited. The paper introduces ARIES and alignment and generation tasks; models struggle with alignment, and GPT-4’s edits often follow wording rather than intent and lack technical detail.
Problem
Datasets for studying how to interpret complex feedback and revise scientific papers are limited, despite scientific writing being a challenging, understudied domain.
Source: §1 · p. 1Method
Results
Models struggle with comment-edit alignment, especially for indirect relationships, while GPT-4 often produces superficially coherent edits that miss intent and technical detail; it scores 27.0 micro-F1 versus human performance of 70.7.
Source: §1 · p. 2Takeaways & Limitations
The paper’s analyses suggest that plausible generated edits can differ substantially from human revisions, and that models may have reasoning gaps worth further study.
Source: §7 · p. 9The study is limited to English AI papers, and its evaluations measure alignment and generation correctness and types rather than impact in real assisted-writing applications.
Source: §9 · p. 10
Abstract
from arXiv · showhide
We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment -- especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4's ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits.
1 Introduction
The paper frames responding to peer feedback as a demanding interpretation-and-editing problem and introduces tasks and ARIES to study it. Models struggle with comment-edit alignment, while GPT-4’s generated edits often miss reviewer intent and technical detail.
1 Introduction
- Interpreting feedback can require deep reasoning because reviewers may express requests indirectly and authors may comply, clarify, or challenge the feedback.The introduction gives the example of requests for a more realistic dataset, phrased as a direct request, criticism, or question.Source: §1 · p. 1
- ARIES links computer-science paper drafts, reviewer feedback, author responses, and revisions, using author responses for silver data and expert labels for a gold test set.The contribution list reports 3.9K reviewer comments automatically matched to edits at 92% precision and a 196-comment human-annotated test set.Source: §1 · pp. 1–2
- The paper formulates comment-edit alignment and edit generation as tasks for studying revisions made in response to reviewer feedback.Alignment identifies existing edits and serves as a stepping stone toward generation, while allowing automatic evaluation.Source: §1 · pp. 1–2
- GPT-4 achieves only 27.0 micro-F1 on comment-edit alignment, versus 70.7 for human performance; generated edits can be coherent but miss intent and technical detail.Indirect comment-edit relationships are especially difficult for models.Source: §1 · p. 2
2 Related work
Prior edit research has addressed stylistic changes, instructions, or edit descriptions, but the paper focuses on contentful scientific-paper edits conditioned on complex reviewer feedback. Related datasets and tools differ in document length, feedback use, or whether they connect comments to specific edits.
2 Related work
- The paper studies contentful edits conditioned on complex feedback in scientific papers, unlike prior work focused on stylistic edits, instructions, or post-hoc edit descriptions.Review comments differ from prior conditioning information in content and linguistic complexity.Source: §2 · p. 2, Figure 2 · p. 3
- ArgRewrite includes contentful teacher feedback, but its essays are much shorter than scientific papers and its comments are not aligned to specific edits.The cited passage compares essays of about 500 words with papers of about 5,000 words.Source: §2 · p. 2
- Scientific-domain revision studies tag edits, support human-in-the-loop editing, or assess writing quality, but do not link peer-review comments to contentful edits.Kuznetsov et al. (2022) separately identify paper edits and align comments to source text without connecting feedback to edits.Source: §2 · pp. 2–3
3 Task Definitions
The paper defines two ways to model revisions that address reviewer feedback: identify relevant edits already made, or generate edits from the comment and source paper.
3 Task Definitions
- An edit is a source paragraph and its corresponding revised paragraph; one side may be null for an addition or deletion.Source: §3 · p. 3
- Comment-edit alignment classifies which candidate edits address a review comment, allowing one comment to map to several edits or several comments to one edit.Evaluation uses edits derived from gold revisions, though candidate edits could also be used.Source: §3 · p. 3
- Edit generation takes a review comment and the original paper text and produces an edit that addresses the feedback and fits the paper’s context.Source: §3 · p. 3
4 Dataset Construction
ARIES combines manually annotated evaluation alignments with a larger synthetic training set of computer-science paper revisions and peer feedback. The construction identifies actionable review comments and links them to paragraph-level paper edits.
4.1 Collecting papers and reviews
- The corpus uses OpenReview papers, reviews, and author responses, comparing the latest pre-review PDF with the latest available revised PDF across 6,501 records.Paper PDFs were parsed with Grobid and S2ORC.Source: §4.1 · p. 4
4.2 Identifying Actionable Feedback
- Actionable feedback is a specific or implied request addressable by a paper edit; annotators extracted 196 comments from 42 reviews.On 10 double-annotated reviews, 88% of spans overlapped, while token-level Jaccard overlap was 65%.Source: §4.2 · p. 4
4.3 Aligning Comments to Edits
- An expert aligned comments to edits using paper versions and author responses, with Cohen’s κ of 0.8 on a 25-comment agreement sample.Although 78% of comments were addressed, 28% were answered only in author responses and 7% of paper edits were not visible in parsed text; 85 comments aligned to textual edits, averaging 2.1 edits each.Source: §4.3 · p. 4
4.4 Creating Synthetic Data
- Synthetic labeling yielded 3,892 high-precision comment-edit alignments by matching quoted review comments and response text to paper edits.The method used a minimum 40-character quote match and at least 25% bigram overlap with response text; it found only 2% of hand-labeled matches, but 46 of 50 sampled alignments were correct.Source: §4.4 · p. 5
5 Comment-Edit Alignment
The paper evaluates models on comment-edit alignment, where systems must identify which paper edits address review comments. Models remain below human performance, with indirect relationships and several distinct error modes posing particular challenges.
5 Comment-Edit Alignment
- Alignment is binary classification over review-comment and candidate-edit pairs, with edits represented as diffs marking additions and deletions.The evaluated approaches include bi-encoders, pairwise and multi-edit cross-encoders, BM25, a biased-random classifier, and an expert human baseline.Source: §5 · p. 5, §5.1 · p. 5
5.2 Results
- GPT-4 methods tend to outperform smaller locally trained models, but no model reaches human performance on comment-edit alignment.The comparison reports precision, recall, and F1, including micro, macro, and addition-only F1; pairwise GPT is evaluated only in the addition-only setting.Source: Table 2 · p. 6, §5.2 · p. 6
- GPT-4 multi-edit’s errors are concentrated: 55% correspond to 20% of comments, and micro-F1 is 30% lower on indirect comments and 24% lower when edits are non-compliant.The authors relate these results to nuanced comments and documents with many edits, including small technical or equation changes.Source: §5.2 · p. 6, §5.4 · p. 7
5.3 False Positives
- In sampled GPT-4 multi-edit false positives, topical similarity accounted for 40%, diff-ignorance 28%, over-generation 28%, and bad parsing 12%.Categories overlap; topical matches can involve unchanged text, diff-ignorance overlooks add/delete markup, and over-generation is specific to the multi-edit format.Source: §5.3 · pp. 6–7
5.4 False Negatives
- False negatives often occur when comment and edit language overlaps little; a code link, for example, may address a request for implementation details.This illustrates why alignment can require recognizing an indirect relationship rather than matching words.Source: §5.4 · p. 7
6 Edit Generation
The section compares GPT-generated edits with human revisions and examines how answerability and edit style affect their performance. GPT often addresses comments comprehensively, but its edits tend to differ from human edits in technical detail and wording.
6.1 Experimental Setup
- The authors generated edits with GPT-4, the best alignment model, instructing it to use placeholders when information was unavailable.They exclude correctness judgments because they may require access to lab notes, raw data, and deep paper-specific expertise.Source: §6.1 · p. 7
6.2 Manual Analysis
- The analysis compares GPT-generated edits with real revisions on 85 comments, using two expert judges and a rubric of edit characteristics.The rubric covers compliance, promises, paraphrasing, and technical details.Source: §6.2 · pp. 7–8
- The judges also assessed comprehensiveness separately from correctness and measured whether comments could be addressed without placeholders or hallucinations.The authors state that generated edits may require information unavailable to the model, such as new experimental results.Source: §6.2 · p. 8
6.3 Results
- GPT edits were comparable to human edits in comprehensiveness overall, with GPT doing better when additional data were unnecessary and humans doing better when they were needed.The analysis also reports that GPT almost always produces coherent, on-topic edits.Source: §6.3 · p. 8, Table 5 · p. 9
- Real edits were judged more technical than GPT edits in 38% of pairs, versus 12% for GPT (p=10−3).GPT often paraphrased the comment, while human edits were more tightly integrated into the paper.Source: §6.3 · p. 8
- 94% of GPT-4 edits were highly compliant, compared with 68% of human edits.The authors note that human authors may instead discuss disagreement on OpenReview without editing the paper.Source: §6.3 · p. 9
7 Conclusion and Future work
The paper introduces comment-edit alignment and edit generation and releases ARIES, a corpus linking scientific-paper reviews to revisions. Its analyses show that plausible generated edits can coexist with difficulty identifying the real comment-edit pairs.
7 Conclusion and Future work
- ARIES contains computer science paper drafts, reviews, author responses, and edits made in response to each review comment.The paper introduces comment-edit alignment and edit generation for scientific-paper revisions based on high-level reviewer feedback.Source: §7 · p. 9
- Table 6 summarizes that GPT edits are more compliant, more likely to paraphrase comments, and less likely to include technical details than human edits.The table reports average compliance and the percentage of examples containing each factor.Source: Table 6 · p. 9
- GPT’s plausible generated edits contrast with its poor alignment performance, suggesting gaps in reasoning about real comment-edit relationships.The authors identify this contrast as a direction for future study.Source: §7 · p. 9
8 Ethics Statement
The ethics statement discusses potential misinformation risks from downstream assistive tools, the public sources used for ARIES, and disclosed AI assistance in the manuscript and experiments.
8 Ethics Statement
- Assistive tools built with ARIES could accelerate dissemination of hallucinations or misinformation, a concern the authors emphasize for scientific text.They urge caution in downstream applications while hoping the work's analysis of model weaknesses encourages caution and better models.Source: §8 · p. 10
- The dataset uses documents from publicly available OpenReview pages, and the authors release their code for reproducibility.Source: §8 · p. 10
8.1 AI Writing Assistance
- An automated feedback system inspired the manuscript, but its feedback was manually evaluated and revised by the authors; GitHub Copilot was used for experiment code.Source: §8.1 · p. 10
9 Limitations
The section sets boundaries on the study, describes paragraph-level edit detection, and analyzes which review requests authors address and what revisions look like.
9 Limitations
- The study covers English-language AI papers, evaluates LLM alignments and generations for correctness and type, and uses some proprietary models whose results may be hard to reproduce if services change.The authors call for future work across more disciplines and languages, and in real assisted-writing applications with user-impact evaluation.Source: §9 · p. 10
- Edit generation lacks the detailed experimental results, code, and lab notes that authors have when revising their papers.The authors describe access to this research-process information as a long-term goal, while noting that collection and efficient LLM use require further work.Source: §9 · p. 10
A Detecting edits
- Paragraph matching scores use bigram overlap and proximity to the previous match; matches below 10% are treated as new paragraphs.The method also merges adjacent paragraphs when doing so raises similarity, to reduce spurious edits from PDF parsing differences.Source: A Detecting edits · p. 12
- Revisions average 40% unchanged paragraphs, 14% minor edits, 14% major edits, 8% deleted paragraphs, and 23% new paragraphs.Minor edits change fewer than 10 tokens and usually fix typos or grammar.Source: A Detecting edits · p. 12
B.1 Comments
- Requests for new baselines, components, or datasets were least likely to be addressed, while term definitions and related-work requests were most often addressed.The analysis covered 182 comments after excluding 7% that did not clearly fit a category; estimates for the most-addressed categories may be high-variance because they were uncommon.Source: §B.1 · p. 13, Table 7 · p. 13
B.2 Edits
- 71% of review-response edits only add a contiguous text span, 34% both add and delete text, and 2% only delete text.An edit adds 89 tokens and deletes 12 on average.Source: §B.2 · p. 13
B.3 Comparison of Synthetic vs. Manually-Annotated Data
The section compares synthetic and manually annotated data, then documents prompt designs for GPT-4 experiments and additional comment annotation statistics.
B.3 Comparison of Synthetic vs. Manually-Annotated Data
- Synthetic and manual data have similar rates of edits adding a new paragraph, but synthetic labels show length-related bias in comment–edit unigram overlap.New-paragraph rates are 64% for manual and 65% for synthetic data; average overlap is 8.3% for synthetic versus 6.6% for manual data, with a 0.4% average difference between length-controlled bins.Source: §B.3 · p. 13
C GPT-4 Prompts
- Pairwise GPT prompts were manually tuned, and evaluation was limited to fully additive edits.Preliminary tests led the authors to place the negative example before the positive one in the one-shot setting.Source: C GPT-4 Prompts · p. 13
C.1 GPT-pairwise (0-shot)
- The zero-shot pairwise prompt asks whether a paragraph added after review addresses a given reviewer comment, with a yes-or-no answer..Source: §C.1 · p. 13
C.2 GPT-pairwise (1-shot)
- The one-shot pairwise prompt gives examples before asking whether an added paragraph addresses the reviewer comment.The prompt presents the comment and paragraph and requests a yes-or-no answer.Source: §C.2 · p. 14
C.3 GPT multi-edit
- The multi-edit prompt requests comment-to-edit ID lists in JSON, allowing many-to-many links and unaddressed comments or edits.Optional model notes were usually uninformative, so the authors did not formally analyze them.Source: §C.3 · p. 14
C.4 GPT edit generation
- The generation prompt asks GPT-4 for a separate reviewer response and a located paper edit, allowing placeholders or reasonable guesses when needed information is unavailable.Separating the response from the edit appeared to reduce edits phrased directly to the reviewer and improve their quality.Source: §C.4 · p. 15
D Additional annotation information
- Among extracted actionable comments, 78% were one sentence, 19% two sentences, and 4% longer than two sentences.The annotation interface allowed arbitrary token spans, though nearly all selected spans aligned roughly with sentence boundaries.Source: D Additional annotation information · p. 15
E Implementation details
The implementation section specifies training and filtering choices, then describes the alignment model families and their scoring or output formats.
E Implementation details
- Alignment training samples 20 negative edits per comment from other training documents, and excludes edited paragraphs shorter than 100 characters.Short paragraphs were excluded because they were often badly parsed equations or text fragments that appeared erroneously as edits.Source: E Implementation details · p. 15
E.1 Alignment model details
- The bi-encoder learns comment and edit embeddings with a triplet loss, while pairwise cross-encoders score comment–edit pairs and BM25 ranks edits by text match.The bi-encoders are DeBERTaV3-large and SPECTER2; cross-encoders include DeBERTaV3-large, LinkBERT, and GPT-4.Source: §E.1 · pp. 15–16
- The GPT-4 multi-edit variant consumes all paper edits and comments together and returns comment IDs linked to edit IDs, splitting long papers into chunks when needed.Decision thresholds are tuned on development data, and a high-recall BM25 filter narrows GPT-4 candidate pairs to reduce cost and evaluation time.Source: §E.1 · p. 16
F Macro-F1 Bias
The section explains why macro-F1 can favor conservative alignment models and reports lower micro-F1 for comment-source alignment than for comment-edit alignment.
F Macro-F1 Bias
- Macro-F1 can favor conservative models because correctly predicting no alignments for a paper with no corresponding edits is scored as 100 F1.A single positive prediction in such a case changes its score to zero, and papers with many positive labels are downweighted by macro averaging.Source: F Macro-F1 Bias · p. 16
- 10.7 vs 18.6 macro-F1 and 10.0 vs 10.8 micro-F1: DeBERTa cross-encoder versus bi-encoder results diverge more under macro averaging.On test pairs, the cross-encoder predicted 17% positives and the bi-encoder 3%, against 1.7% true positives.Source: F Macro-F1 Bias · p. 16
G Comment-source alignment
- Comment-source alignment is more ambiguous than comment-edit alignment because multiple source locations may address a comment, but only the real edit location counts as correct.The task omits newly added paragraphs and excludes GPT-4 due to its high cost.Source: G Comment-source alignment · pp. 16–17
G.1 Results
- Comment-source alignment models have substantially lower micro-F1 than comment-edit alignment, though the results are not directly comparable because the dataset split differs.The higher macro scores may reflect models’ tendency to predict negatives and gain more perfect scores on comments without aligned edits.Source: §G.1 · p. 17, Table 8 · p. 17