PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

Yaochen Wang, Chenguang Wang, Ming Li, Haomin Zhuang, Xinyue Zeng, Han Chen, Kwesi Adu Cobbina, Dang Nguyen, Dianqi Li, Dawei Zhou, Tianyi Zhou

arXiv:2611.02244v1cs.CL

TL;DR

Whether AI review scores reflect authors’ improvements through revision remains unclear. Using PaperTrajectory-2026 and independent assessments from eight AI reviewers, the paper finds that most major improvements receive unchanged scores, while negative user opinions can erase or reverse score advantages.

  • Problem

    It remains unclear whether AI review scores reflect improvements authors make through revision.

    Source: §1 · p. 1

  • Method

    The study uses PaperTrajectory-2026’s revision annotations and separate quality judgments, alongside independent assessments of anonymized versions by eight AI reviewers.

    Source: §1 · p. 2

  • Results

    Across eight AI reviewers, most major improvements receive unchanged scores, revision dimensions provide no consistent guide to score increases, and negative user opinions can erase or reverse score advantages.

    Source: §1 · pp. 2–3, §8 · p. 12

  • Takeaways & Limitations

    AI reviewers should recognize substantive improvements and remain stable when user preferences change without new scientific evidence.

    Source: §8 · p. 12

Abstract

from arXiv · show

AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using PaperTrajectory-2026, the 2026 release of PaperTrajectory, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes.

1 Introduction

The paper asks whether AI review scores reflect authors’ substantive revisions, a question not settled by prior work on review quality, rewriting, or feedback-driven revision. It introduces PaperTrajectory-2026 and uses independent assessments from eight AI reviewers to examine score changes and sensitivity to user opinions.

1 Introduction

  • Prior studies examine AI-review feedback and score changes under rewriting or rhetorical changes, but do not establish how scores respond to added scientific evidence or substantive improvement.Source: §1 · p. 1
  • PaperTrajectory-2026 reconstructs real revision trajectories with 14-dimensional change annotations, separate overall-quality judgments, supporting evidence, and traceable version records.Source: §1 · pp. 2–3
  • Five graduate-level researchers validated sampled annotations, while eight AI reviewers independently assessed anonymized manuscript versions to test score responses to revision.The paper also tests associations between revision dimensions and score changes, and whether user opinions can outweigh observed revision gains.Source: §1 · pp. 2–3
  • Most major improvements receive unchanged scores across reviewers, so substantive content improvements do not reliably yield higher ratings.The paper also finds that a negative user opinion can erase or reverse a revised paper’s score advantage, with sensitivity varying across reviewers.Source: §1 · pp. 2–3
  • Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers.Source: §1 · pp. 2–3

2 Related Work

Prior work studies manuscript revision, AI-generated feedback, and paper assessment, but the relationship between substantive revision and AI review scores remains underexplored.

2 Related Work

  • Existing revision resources align successive versions, link edits to reviews and author responses, or evaluate feedback-driven manuscript and experiment changes.Source: §2 · p. 3
  • Evaluation research examines LLM judging, review deficiencies, limitation identification, human–model differences, and methods for ranking or estimating paper quality.Source: §2 · p. 3
  • The specific gap is whether AI review scores reflect substantive scientific improvements across successive versions of the same paper.Source: §2 · p. 3

3 PAPERTRAJECTORY

PaperTrajectory-2026 is a dataset of revision trajectories for accepted papers at five AI conferences, annotated for revision content and overall quality change.

3.1 Dataset Construction

  • The construction retains each paper’s earliest version and then the first subsequent version more than 30 days later, pairing adjacent retained versions.Of 9,092 pairs, 9,066 have completed annotations; the remainder are recorded as missing.Source: §3.1 · p. 3

3 PAPERTRAJECTORY

  • The 2026 dataset covers 6,959 papers, 16,051 versions, and 9,092 transitions from ICLR, ICML, ACL, CVPR, and ECCV.Source: §3 · p. 3, Figure 2 · p. 4

3.2 Revision Taxonomy

  • The 14-dimensional taxonomy describes change across methods, evidence, theory, claims, disclosure, positioning, and presentation, while a separate label assesses overall quality change.Source: §3.2 · pp. 3–4

3.3 Revision Annotation and Human Validation

  • GPT-6 Astra annotates complete version pairs and overall quality separately; human evaluators marked 98.6% of change descriptions accurate and confirmed improvement in 98.7% of sampled major-improvement pairs.Subsequent analyses use the automatic annotations.Source: §3.3 · p. 4

4 How Do Accepted Papers Evolve Through Revision?

Across accepted-paper trajectories, revisions emphasize empirical support and presentation, major improvements tend to span evidence and framing, and later revisions become narrower.

4.1 What Do Authors Change?

  • Results, visuals, ablations, and evaluation protocols change substantially in 43–59% of revisions, while methods change substantially in only 10.2% and writing changes are mostly minor.The contrast between results and methods holds across conferences, trajectory lengths, and gaps between versions.Source: §4.1 · p. 5

4.2 What Distinguishes Major Improvements?

  • 44.5% of revisions with substantial changes to both evidence and framing receive a major-improvement label, compared with 14–17% when only one changes and 1.0% when neither changes.Major improvements substantially revise 7.9 dimensions on average, compared with 3.1 for minor improvements.Source: §4.2 · p. 5

4.3 How Does Revision Change Along a Trajectory?

  • Among 1,712 papers with at least two revisions, the mean number of substantially changed dimensions falls from 5.6 in the first pair to 2.9 in the last, while major-improvement labels decline from 22.5% to 11.2%.These patterns hold within each conference and trajectory-length group.Source: §4.3 · pp. 5–6

5 Do Major Paper Improvements Lead to Higher AI Review Scores?

Major improvements usually leave AI review scores unchanged, even when human evaluators confirm the improvement. Unchanged scores are not explained by a rating ceiling and do not by themselves show that reviewers failed to notice the changes.

5 Do Major Paper Improvements Lead to Higher AI Review Scores?

  • 55–78% of major-improvement revisions retain the earlier overall assessment across reviewers, and unchanged scores remain the majority among human-confirmed improvements.The human-confirmed subset includes clear or slight improvements.Source: §5 · pp. 6–7, Figure 4 · p. 6
  • Excluding revisions that start at the maximum rating changes each model’s unchanged-score rate by less than 0.5 percentage points.For ICLR, Astra often keeps an OA of 3 after revision, well below the maximum of 10.Source: §5 · p. 6
  • Astra’s review of Cautious Optimizers acknowledges new experiments as strengths while retaining its OA of 3 over remaining concerns.The example shows that an unchanged overall assessment does not establish that a reviewer failed to recognize changes.Source: §5 · pp. 6–7

6 Which Revisions Are Associated with Higher AI Review Scores?

Across reviewers, score changes show little agreement, dimension–score associations vary, and revision indicators offer only limited predictive guidance for new papers.

6.1 Do Reviewers Reward the Same Revisions?

  • Only seven of 1,414 revisions receive higher scores from all eight reviewers, and pairwise Cohen’s κ ranges from −0.071 to 0.170.For 385 revisions, at least one reviewer raises the score while another lowers it; weak agreement does not mean reviewers reward entirely different revisions.Source: §6.1 · p. 7, Table 1 · p. 7
  • The seven unanimous increases fall within the two-to-nine range from rating-conditioned permutations, so they do not establish excess agreement beyond reviewers’ scoring patterns.Initial ratings matter because score-increase rates vary with the earlier rating.Source: §6.1 · p. 7, Figure 5 · p. 8

6.2 Which Revision Dimensions Are Associated with Score Increases?

  • Methods changes associate with more score increases for GPT-5.5 and Gemini 3.5, while Statistics associates with more for GPT-5.5 and Sonnet but fewer for Astra.No dimension has a positive association surviving multiple-testing correction for all eight reviewers; Methods, Coverage, and Statistics show corrected cross-reviewer differences.Source: §6.2 · pp. 8–9, Table 2 · p. 9
  • AUCs of 0.51–0.64 and relative Brier-loss reductions of −1.6% to 3.3% indicate modest ranking signal but little probability-accuracy gain from revision indicators.The Brier-loss range includes worse performance than the baseline that predicts each reviewer’s training-fold score-increase rate.Source: §6.2 · pp. 9–10, Figure 6 · p. 9

7 Can User Opinions Outweigh Revision Gains?

The study tests whether user opinions without new scientific evidence can change the advantage a revision earned under neutral review. Criticism often erases or reverses that advantage, with outcomes varying across reviewers.

7.1 Criticism of the Revised Manuscript

  • Under criticism of the revised version, GPT-5.5 scores it lower than the earlier version in 54.3% of cases and ties in another 43.5%.Sonnet and Haiku erase or reverse more than 95% of their initial advantages; for Sol and GLM, ties are more common than reversals.Source: §7.1 · p. 10, Figure 7 · p. 10

7.2 Opposing Opinions about the Two Versions

  • With opposing opinions, Sol’s reversal rate rises from 11.3% under criticism alone to 74.6%, while GPT-5.5, Sonnet, and Haiku reverse about nine in ten or more initial advantages.The comparison reuses the revised version’s score under criticism, so the difference for Sol comes from praising the earlier version.Source: §7.2 · pp. 10–11, Figure 7 · p. 10

7.3 Revision Sensitivity and Opinion Stability

  • Astra reverses only 7.1% of initial advantages under opposing opinions, but 45.8% become ties; its low reversal rate does not mean every advantage survives.Under criticism alone, Astra keeps the revised version ahead in about three-quarters of cases.Source: Figure 7 · p. 10, §7.2 · p. 11, §7.3 · p. 11
  • Even larger initial advantages can reverse under opposing opinions: 92.2% of GPT-5.5’s and 87.5% of Haiku’s gains reverse in that group.Every reviewer has a lower estimated reversal rate for larger gains, but the groups differ in papers and conference composition and have wider uncertainty intervals.Source: §7.3 · p. 11, Figure 8 · p. 11
  • Shared criticism narrows the average score gap for GPT-5.5, Sol, and both Claude reviewers, widens it for both Gemini models, and has a near-zero estimated effect for Astra.The response to criticism can depend on the manuscript version as well as the reviewer.Source: §7.3 · p. 12

8 Conclusion

The conclusion draws together findings on revision trajectories, AI reviewers’ score responses, and vulnerability to user opinions. It calls for assessments that connect manuscript evidence to overall judgments and remain stable when scientific evidence is unchanged.

8 Conclusion

  • Major improvements often strengthen empirical evidence and framing, while later revisions tend to become more focused.Source: §8 · p. 12
  • Across eight AI reviewers, most major improvements receive unchanged scores, and revision dimensions provide no consistent guide to score increases.Reviewers also disagree on which revisions deserve higher scores.Source: §8 · p. 12
  • A single unsupported user opinion can erase or reverse a revised paper’s score advantage, motivating stable judgments when no scientific evidence changes.Source: §8 · p. 12

A.1 LLM-Based Evaluation

AI-review scores depend on evaluation setup, while existing revision resources do not establish whether combined manuscript changes improve overall quality. This study connects naturally occurring revision trajectories, dimension-level changes, and overall-quality judgments to independent AI-review assessments.

A.1 LLM-Based Evaluation

  • Rubric design, score labels, and prompt complexity can affect AI judgments, while model self-preference and user opinions can also shape responses.Prior work motivates separating responses to the assessed manuscript from responses to surrounding instructions or opinions.Source: §A.1 · p. 17

A.2 Paper Revision and Improvement

  • Existing revision datasets expose local edits, intentions, or review-linked changes, but do not establish whether combined manuscript changes constitute an overall quality improvement.PaperTrajectory distinguishes dimension-specific change from overall quality rather than treating revision magnitude or chronology as improvement.Source: §A.2 · p. 17

A.3 AI-Based Scientific Reviewing

  • Prior AI-review studies examine critique, human alignment, or responses to controlled changes; this study connects natural revision trajectories with independent scores and user-opinion sensitivity.The cited work also examines claim-evidence grounding, critique, and review-protocol design.Source: §A.3 · p. 18

B.1 Paper Collection and arXiv Matching

  • The collection uses accepted-paper lists from five conferences, manually matched to arXiv entries, and retains version numbers and timestamps as a snapshot of public histories.The snapshot does not record internal submission or review stages.Source: §B.1 · p. 19

B.2 Version Selection and Trajectory Construction

  • Trajectory construction retains the earliest subsequent version more than 30 days after the last retained version, then compares adjacent retained versions.Papers need at least two retained versions; exactly 30 days does not qualify, and intermediate versions may be skipped.Source: §B.2 · p. 19

B.3 Full Revision Taxonomy

  • The 14-dimensional taxonomy separates changes in methods, empirical evidence, claims, discussion, disclosure, literature, writing, and visuals.It distinguishes, for example, concrete results from changed evaluation protocols and writing from accompanying scientific changes.Source: §B.3 · pp. 19–20

B.4 Annotation Instructions and Quality Labels

  • Dimension labels record change magnitude, while a separate overall-quality judgment weighs benefits against harms rather than aggregating dimension labels.More experiments, higher reported numbers, stronger claims, or extensive rewriting alone do not establish net improvement.Source: §B.4 · p. 20

B.5 Human Validation

Human evaluators largely confirmed the displayed change descriptions and major-improvement labels, while agreeing more often on whether improvement occurred than on its degree. The study then uses independent model assessments of anonymized manuscript versions under neutral and opinion-conditioned prompts.

B.5 Human Validation

  • Evaluators rated 2,760 of 2,800 change descriptions accurate (98.6%), and confirmed improvement for 99 of 100 major-labeled assessments.When every assigned evaluator had to confirm improvement, 74 of 75 major-labeled pairs (98.7%) were confirmed.Source: §B.5 · p. 21
  • Exact overall-category agreement on duplicated pairs was 62.0% (ordinal Krippendorff’s α = 0.521), rising to 88.0% for distinguishing improvement from no improvement.Evaluators agreed more often on improvement’s presence than on its degree.Source: §B.5 · p. 21

C Experimental Details

  • Eight reviewer models assess each anonymized manuscript independently, without the paired version, revision history, or annotations; opinion conditions add user opinion and acceptance preference while keeping manuscript and criteria fixed.The sample contains 1,484 automatically labeled major-improvement pairs with both endpoints anonymized.Source: C Experimental Details · p. 22

D.1 Overall Quality by Conference; D.2 Composition of Major Improvements

  • The appendix tables report overall-quality judgments by conference and pooled, and describe change magnitudes by dimension and overall-quality label.Uncertain judgments remain in the denominator for the conference distribution.Source: Table 5 · p. 23, Table 6 · p. 23

D.3 First-to-Last Changes and Stratified Results

Comparing first and last transitions shows fewer major improvements and fewer dimensions with major changes at the end of trajectories. The appendix supplies stratified comparisons and describes the whole-paper bootstrap procedure used for uncertainty intervals.

D.3 First-to-Last Changes and Stratified Results

  • −11.33 percentage points is the last-minus-first difference in major-improvement rate, and −2.72 is the difference in mean dimensions with major change.The 95% percentile intervals are [−13.79, −8.76] and [−2.95, −2.48], respectively; the comparison uses 5,000 whole-paper bootstrap resamples.Source: §D.3 · p. 23, Table 9 · p. 25
  • The appendix reports co-occurring major-change frequencies and stratifies revision patterns by results, methods, evidence, and framing changes.Its trajectory-length and conference comparisons report first-to-last rates and mean numbers of major-change dimensions.Source: Table 7 · p. 24, Table 8 · p. 24, Table 9 · p. 25

E.1 Uncertainty Estimation

  • Uncertainty intervals use 3,000 whole-paper bootstrap resamples, retaining all eligible transitions for each sampled paper and resampling within conference for conference-specific estimates.The intervals quantify across-paper variation for the recorded assessments.Source: §E.1 · p. 26

E.2 Full Estimates and Common-Sample Check

Most reviewers leave scores unchanged on major improvements, and their directional agreement is only slightly above a chance reference. Revision-dimension associations vary across reviewers, while sparse support constrains the coefficient analysis.

E.2 Full Estimates and Common-Sample Check

  • Unchanged scores remain the majority outcome for every reviewer on the common sample of 1,414 revisions assessed by all eight models.The appendix provides full-sample and common-sample estimates with uncertainty intervals.Source: §E.3 · p. 26, §E.2 · p. 26, Table 11 · p. 26

E.3 Human-Confirmed Improvements

  • Unchanged scores also remain the majority outcome for each reviewer on the 60 human-confirmed improvements (Haiku covers 59).Intervals use paper resampling within conference.Source: §E.3 · p. 26, Table 12 · p. 27

E.4 Conference-Specific Results

  • Conference-specific tables report higher, same, and lower neutral-score directions using each conference’s original rating categories.The supplied appendix passage identifies the table scope but does not state conference-specific outcomes.Source: §E.4 · p. 26, Table 13 · p. 27

E.5 Initial Ratings and Ceiling Checks

  • The appendix tabulates initial-rating concentration, with modal overall assessment defined as the most frequent earlier-version rating.It also reports how often ties occur at that initial modal rating.Source: §E.5 · p. 28, Table 14 · p. 28

F.1 Directional Agreement Diagnostics

  • Across all eight reviewers, mean directional agreement is 53.49% versus a 50.37% chance reference, with Fleiss’ κ = 0.063.Pairwise agreement ranges from 45.12% to 63.30%.Source: §F.1 · p. 29

F.2 Conditional Permutation Comparisons

  • Conditioning on each reviewer’s exact initial rating raises the expected unanimous-increase count to 4.84, placing the seven observed unanimous increases within the [2, 9] reference range.The conditioned comparison has p = .178; the full reviewer-count distribution still differs from that reference (global p = .00020).Source: §F.2 · p. 29

F.3 Statistical Model and Inference

  • The regression models predict higher versus unchanged-or-lower scores from revision indicators, with each coefficient conditional on the other included dimensions.The coefficient analysis uses six dimensions with at least 100 papers in both change groups for every reviewer; other sparse dimensions remain in regularized prediction.Source: §F.3 · pp. 29–30

F.4 Full Association Estimates

  • Reviewer-dimension associations differ: Methods, Coverage, and Statistics show supported cross-reviewer differences after adjustment, while the six-dimension feature subset was selected after sparsity diagnostics.The subset-selection process is not accounted for by the reported multiplicity adjustment.Source: §F.4 · pp. 30–31

F.5 Additional Sensitivity Analysis

Sensitivity analyses find that GPT-5.5’s positive Methods association persists under sample restrictions and a penalized-logit diagnostic, while its Statistics association does not remain significant in the earliest-revision sample. The appendices also detail held-out prediction procedures and measures of opinion-driven score reversals, noting uncertainty from single reviews and positive-gap selection.

F.5 Additional Sensitivity Analysis

  • GPT-5.5’s Methods association persists under both sample restrictions and the nine-dimension Firth-logit diagnostic, while its Statistics association loses adjusted significance in the earliest-revision sample.The Methods odds ratio in the diagnostic is 1.575 [1.240, 2.001] (q = 0.0021); extreme estimates for some models reflect sparse joint patterns.Source: §F.5 · p. 32

F.6 Held-Out Prediction Details

  • Held-out prediction uses separate L2-regularized classifiers on twelve revision dimensions, evaluated with shared paper-grouped folds and compared against a training-prevalence baseline.Bootstrap intervals keep predictions and folds fixed, excluding refitting and fold-assignment variability; they are pointwise and unadjusted for multiple comparisons.Source: §F.6 · pp. 32–33, Table 19 · p. 32, Table 20 · p. 33

G.1 Estimators and Uncertainty

  • The opposing-opinion rates compare revised scores under criticism with neutral or praise-based scores, using revisions with a positive neutral score gap for their denominator.Rates weight eligible revisions equally within the observed conference composition, and uncertainty intervals reflect whole-paper resampling.Source: §G.1 · p. 34
  • Because each manuscript is reviewed once per condition, positive neutral gaps may partly reflect unusually high review scores that would shrink on another neutral review.Repeated neutral reviews would help determine how much additional instability is associated with user opinions.Source: §G.1 · p. 34

G.2 Full Ordering-Change Estimates

  • Table 21 reports opinion-induced ordering-change rates, sample sizes, and pointwise whole-paper bootstrap intervals; erase-or-reverse rates include ties.The rate denominator is the count of revisions with a positive neutral gap.Source: §G.2 · p. 34, Table 21 · p. 34

G.3 Common-Sample and Additional GPT Checks

Matched-sample checks preserve large reviewer differences in opinion-driven reversals, while analyses of initial score advantages and shared criticism qualify how those changes are measured. The annotation and review prompts specify separate overall-quality judgments, a fourteen-dimension rubric, and conference-specific rating criteria.

G.3 Common-Sample and Additional GPT Checks

  • 7.2% (Astra) to 97.6% (GPT-5.5): opposing-opinion reversal rates span widely across reviewers in matched samples, with reviewer ranking unchanged.A separate medium-reasoning GPT collection yielded 97.5%, 75.8%, and 6.6% reversals for GPT-5.5, Sol, and Astra; it does not isolate reasoning-effort effects.Source: §G.3 · pp. 34–35, Table 22 · p. 35

G.4 Initial Revision Advantage and Opinion Stability

  • Larger-advantage groups contain 26–175 revisions versus 216–549 for one-category gains; these descriptive contrasts do not establish a causal effect of initial advantage.The groups are not adjusted for conference composition or other manuscript differences.Source: §G.4 · pp. 35–36

G.5 Revision Gaps Under Shared Framing

  • Shared criticism narrows mean score gaps by 4.90 percentage points for GPT-5.5 and 3.17 for Sol, while both Gemini models show larger gaps.Sonnet and Haiku show reductions of 1.83 and 2.28 points, respectively, with intervals excluding zero; all complete observations, including ties and decreases, are included.Source: §G.5 · p. 36, Table 24 · p. 36

G.6 Conference-Specific Results

  • Conference-specific conditional rates are reported alongside denominators based on revisions with complete opinion ratings and a positive neutral gap.The supplied passages identify the table’s scope but do not provide its conference-level rate values.Source: §G.6 · p. 36, Table 25 · p. 37

H Complete Annotation and Review Prompts

  • The appendix reproduces the annotation and review prompts, with manuscript placeholders filled separately for each assessment.The review prompt requests strengths, weaknesses, and an overall rating.Source: H Complete Annotation and Review Prompts · p. 38

H.1 Revision Annotation Instructions and Output Format

  • The annotation rubric judges overall quality separately from change dimensions, weighs benefits against harms, and defines major improvement as clear evidence of substantial overall-quality improvement.The instructions say not to decide improvement by counting or voting across dimensions.Source: §H.1 · pp. 38–39

H.3 Conference-Specific System Prompts

  • The ICLR profile adopts the overall-rating rubric from Li et al. (2026b).Source: §H.3 · p. 41

Related papers

Loading 2611.02244v1…