PaperlayerView exact revision on arXiv

Source-linked AI summary

AppliedScientist: Automated Scientific Revision Through Iterative AI Reviewing

Vidushee Vats, Karun Sharma, Shengzhi Li, Shichao Pei

arXiv:2609.14738v2cs.AI

TL;DR

Automated-review systems are useful only if acting on their feedback measurably improves papers, but whether AI reviewers can repeatedly guide revisions to existing work remains unclear. AppliedScientist pairs an autonomous scientist with an independent reviewer and compares reviewer-guided revisions with fixed-prompt self-revision. Reviewer guidance improves more, while execution weaknesses are resolved far more often than idea weaknesses.

  • Problem

    Whether an independently evaluated AI reviewer can repeatedly guide code, experiment, and manuscript revisions of the same existing paper remains unclear.

    Source: Introduction · p. 1

  • Method

    AppliedScientist revises existing papers through a scientist that uses revision history and a reviewer that independently evaluates each version and supplies feedback.

    Source: Methodology · p. 3

  • Results

    Reviewer-guided revision improves more than fixed-prompt self-revision; 128 of 150 execution weaknesses (85.3%) but only 2 of 18 idea weaknesses (∼11.1%) are resolved.

    Source: Results and Discussion · pp. 5–6

  • Takeaways & Limitations

    Iterative revision can strengthen execution and evidence in an existing contribution, but concerns about novelty and significance generally require returning to ideation.

    Source: Results and Discussion · p. 6, Conclusion · p. 6

    Running the complete revision process for one paper costs approximately $65, limiting the number of papers evaluated.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Automated reviewing systems are increasingly evaluated based on the quality of the reviews they produce. Yet a review is only useful if acting on it leads to a measurable improvement in the paper. We present AppliedScientist, a closed-loop system that couples an autonomous AI scientist with an AI reviewer, and evaluate it by iteratively revising rejected papers from a range of research subfields. To mirror how human authors build on earlier drafts, the AI scientist has access to its previous versions during revision. To avoid bias from prior judgments, however, each review is generated independently, with the reviewer having no memory of earlier feedback or scores. We compare three revision settings: one initialized with the original venue reviews, one initialized with AI-generated reviews, and autonomous self-revision using the same fixed prompt in every round. Because the reviewer both guides and evaluates the revision, we also assess the human-initialized revisions using Stanford Reviewer as an independent evaluator. Reviewer-guided revision consistently improves more than fixed-prompt self-revision, and Stanford Reviewer also assigns higher scores to later revisions. AppliedScientist resolves 128 of 150 execution-related weaknesses (85.3%), but only 2 of 18 idea-related weaknesses (11.1%), suggesting that iterative revision is effective at improving experiments and implementation, but rarely changes concerns about novelty or significance.

Introduction

AppliedScientist asks whether an AI reviewer can repeatedly guide revisions to an existing paper without replacing its central contribution. It couples a revising scientist with fresh reviewer feedback, and reviewer-guided revisions outperform fixed-prompt self-revision, especially on execution weaknesses.

Introduction

  • AppliedScientist addresses whether an independently evaluated AI reviewer can guide code, experiment, and manuscript revisions of the same paper without replacing its contribution.Authors may need to reproduce results, run new experiments, and assess stronger baselines to address reviewer feedback.Source: Introduction · p. 1
  • The system revises an existing implementation, runs experiments, analyzes results, and updates the manuscript using fresh reviews of each version.The scientist retains prior versions and feedback, while the reviewer assesses each current version independently.Source: Introduction · p. 1
  • Three guidance settings compare original venue reviews, AI-generated initial reviews, and a fixed self-review prompt; Stanford Reviewer independently evaluates human-initialized trajectories.The scientist has the same research capabilities in all three conditions, isolating the effect of guidance.Source: Introduction · p. 1
  • Reviewer-guided revision improves more than autonomous self-revision, resolving 128 of 150 execution weaknesses (85.3%) but only 2 of 18 idea weaknesses (∼11.1%).The results suggest revision strengthens implementation and experimental evidence more readily than novelty or significance.Source: Introduction · p. 2
  • The paper contributes a revision system, evaluates reviewer guidance against fixed-prompt self-revision, and characterizes which weaknesses autonomous revision can resolve.Execution concerns are more tractable than novelty-related concerns.Source: Introduction · p. 2

Related Work

Prior work has automated scientific research, peer review, and manuscript revision, but often addresses only part of the revision process. AppliedScientist focuses on repeated review-guided changes to an existing paper, including experimental validation.

AI Scientists and Autonomous Research

  • Prior AI-scientist systems cover research stages from idea generation to peer review, while AppliedScientist focuses on iterative revision guided by repeated peer review.Earlier systems use review to archive evaluations, refine figures, or construct training preferences rather than guide this revision setting.Source: Related Work · p. 2

Automated Peer Review

  • Automated peer-review work primarily evaluates reviews themselves; this work separately benchmarks its reviewer and tests whether acting on its feedback improves research.Prior systems ground critiques through expert-review fine-tuning, human-preference rewards, claim tracing, multiple agents, personas, or retrieval.Source: Related Work · p. 2

AI-Assisted Scientific Revision

  • Existing revision systems often limit the action space to text or presentation, use synthetic manuscripts, or provide fixed feedback rather than repeated reviews of revised papers.AppliedScientist instead studies iterative revision that can include experiment execution and validation of feedback.Source: Related Work · p. 2

Methodology

AppliedScientist couples a scientist that revises an existing paper with a reviewer that assesses each version independently. The scientist can act on accumulated history and feedback, while the reviewer supplies fresh guidance in reviewer-guided conditions.

AppliedScientist

  • The scientist revises an existing contribution rather than generating a new research idea, and records concerns as unresolved when addressing them would require changing that contribution.Depending on feedback, revisions can include code changes, reproduced results, added baselines or ablations, experiments, analysis, and manuscript updates.Source: Methodology · pp. 2–3
  • The scientist verifies reviewer suggestions, runs experiments, analyzes results, and updates the manuscript; the reviewer independently assesses each revision and returns structured feedback.The feedback guides the following round in reviewer-guided revision.Source: Figure 1 · p. 3

Revision Loop

  • The scientist retains prior versions and intermediate results so revisions build on earlier decisions, while the reviewer independently evaluates each version without earlier drafts, comments, or scores.Independent reviews are intended to avoid bias from prior judgments or scores.Source: Methodology · p. 3

Review Component

  • The reviewer identifies contributions, methodology, and evaluation setting, searches literature beyond cited references, then assesses novelty, positioning, evaluation adequacy, and impact.It synthesizes findings into a structured review with strengths, weaknesses, and suggestions for the next revision.Source: Methodology · p. 3

Revision Loop

  • At each round, the scientist uses the prior manuscript, code, revision history, and guidance to produce updated code, experimental artifacts, and manuscript; the reviewer evaluates only the current manuscript.In reviewer-guided conditions, the new review guides the next round; in self-revision, a fixed prompt remains the guidance and reviews serve only for evaluation.Source: Methodology · p. 3

Data

  • The evaluation set contains 30 ICLR papers spanning five research domains, selected with publicly available code that can be reproduced on a single GPU within the VRAM budget.The human-initialized trajectory starts with venue reviews, after which fresh AI-reviewer feedback guides successive revisions.Source: Table 1 · p. 4, Figure 2 · p. 4

Experimental Setup

The evaluation uses 30 ICLR papers and compares three sources of revision guidance. The reviewer also scores saved versions, with Stanford Reviewer providing an external assessment of the human-initialized trajectories.

Data

  • The study mines 25 rejected and 5 borderline-accepted ICLR papers with public reviews and code, spanning research domains and executable on CPU or a single GPU with 96 GB VRAM.Sampling preferred papers with low variance in original human reviewer scores.Source: Experimental Setup · p. 4

Setup

  • Human-initialized revision starts with venue reviews, AI-initialized revision starts with AI reviews, and autonomous self-revision uses the same fixed prompt throughout.The scientist has the same capabilities across conditions, isolating the effect of guidance; each run takes about nine hours on a 96 GB VRAM GPU.Source: Experimental Setup · p. 4

Evaluation

  • The reviewer scores every saved version, while Stanford Reviewer independently evaluates only the human-initialized trajectories and provides no feedback to the scientist.Score and threshold analyses use the 30 completed human-initialized trajectories.Source: Experimental Setup · p. 4

Reviewer Evaluation

The reviewer is evaluated against human ratings and venue decisions, and compared with existing automated reviewers. It aligns with venue judgments and outperforms baselines on nearly every review dimension, with comparable performance to Stanford Agent Reviewer on Constructive Value.

Pairwise reviewer evaluation

  • The reviewer outperforms all baselines across nearly every evaluation dimension, but its Constructive Value win rate is comparable to Stanford Agent Reviewer’s.Constructive Value win rates are 44.3% versus 45.7%, with 10% of comparisons tied.Source: Reviewer Evaluation · p. 4

Evaluation setup

  • A search-enabled judge compares anonymized reviews across technical accuracy, constructive value, analytical depth, and significance assessment, penalizing hallucinations and generic feedback.The evaluation also assesses reviewer alignment with venue ratings and decisions on recent ICLR papers.Source: Reviewer Evaluation · p. 4

Score alignment

  • ρ = 0.571 and 76.2% venue-decision agreement with DeepSeek V4 Flash, versus ρ = 0.48 and 67.2% with MiniMax M2.7; Stanford Reviewer scores ρ = 0.51 and 71.1%.Figure 3 reports Spearman correlation with mean human ratings and agreement with the venue’s accept/reject decision.Source: Reviewer Evaluation · p. 5, Figure 3 · p. 5

Results and Discussion

Reviewer-guided revision improves papers more than fixed-prompt self-revision, and AI-generated initial feedback reaches nearly the human-initialized condition’s final score. Improvements are strongest for execution weaknesses; novelty and significance concerns rarely change.

AI-initialized revision

  • AI-initialized revision consistently outperforms autonomous self-revision and converges to nearly the human-initialized condition’s final score after five rounds.The AI Reviewer supplies the initial feedback on V0 in this condition.Source: Results and Discussion · p. 5

Domain coverage

  • The upward score pattern appears in every represented scientific domain, rather than being confined to one research area.Figure 2 presents trajectories for the human-initialized condition across domains.Source: Results and Discussion · p. 5

External evaluation

  • Stanford Reviewer independently observes an upward trajectory across human-initialized versions, although its absolute scores differ from the paper’s reviewer.It evaluates saved human-initialized versions only; its reviews are not provided to the scientist.Source: Results and Discussion · p. 5

Score improvements

  • The reviewer’s average score rises from 5.37 on original papers to 6.53 at each paper’s best revision, a gain of +1.16 points; Stanford Reviewer rises from 5.50 to 6.15, a gain of +0.65 points.At their best revision, all 30 papers score at least 6 under the reviewer, compared with 12 at V0.Source: Results and Discussion · p. 5, Table 4 · p. 6

Reviewer comparisons

  • Table 3 compares anonymized reviews from the AI reviewer and automated baselines using a search-enabled judge; Stanford Reviewer is evaluated on 100 papers.The table reports win, loss, and tie percentages.Source: Table 3 · p. 6

Rejection reasons

  • Among sampled rejected papers, 48% involve execution issues, 35% idea-related issues, and the remaining 17% involve both.The classification uses a random sample of 500 rejected ICLR papers, separate from the revision evaluation set.Source: Results and Discussion · p. 6

Revision outcomes

  • 128 of 150 execution weaknesses (85.3%) are resolved, versus 2 of 18 idea weaknesses (∼11.1%); reviewer-guided revision also outperforms fixed-prompt self-revision, while AI-initialized revisions nearly match human-initialized scores after five rounds.Revision succeeds more often on requests for stronger evidence or clearer presentation than on novelty or significance judgments.Source: Results and Discussion · pp. 5–6, Figure 4 · p. 5

Limits of revision

  • Novelty objections persist through revision rounds, and additional evidence seldom changes the outcome once reviewers judge the contribution insufficiently novel.The authors suggest that improving the idea itself requires returning to ideation rather than continuing the revision loop.Source: Results and Discussion · p. 6

Conclusion

Across 30 ICLR papers, reviewer-guided revision improved papers more than fixed-prompt self-revision, with gains concentrated in execution rather than changes to the research idea. Evaluation cost constrained how many papers could be studied.

Conclusion

  • Across 30 ICLR papers, reviewer-guided revision consistently improved papers more than autonomous self-revision with a fixed prompt.Most improvements involved experiments, implementation, evaluation, and presentation rather than changes to the underlying research idea.Source: Conclusion · p. 6
  • Only 2 of 18 idea-related weaknesses (11.1%) were addressed, whereas most empirical weaknesses were resolved.The authors conclude that autonomous revision can strengthen execution of existing research, while novelty and significance concerns generally require returning to ideation.Source: Figure 5 · p. 7, Conclusion · p. 7
  • Running the complete revision process for one paper costs approximately $65, limiting the number of papers evaluated.Source: Conclusion · p. 7

Supplementary Material AppliedScientist: Automated Scientific Revision

The supplementary material describes Deep Search and Retrieval, including hybrid paper search, query tools, safeguards against future-work criticism, and targeted full-paper analysis. It also lists further materials on the reviewer, revision process, and prompts.

Supplementary Material AppliedScientist: Automated Scientific Revision

  • The supplementary contents also list sections on reviewer evaluation, PDF input representation, revision rounds, example trajectories, per-domain results, novelty objections, computation cost, and prompts.Source: Supplementary Material… · p. 9

A.1 Deep Search and Retrieval

  • The index combines title-and-abstract embeddings with BM25 lexical indexes, then fuses batched multiquery retrieval and expands related work from a known arXiv identifier.The index covers over one million arXiv papers and uses cross-query deduplication.Source: §A.1 · p. 9
  • The reviewer exposes batch, related, and query commands, and uses CLI usage documentation rather than an HTTP API to avoid invented parameter formats.Source: §A.1 · p. 9
  • Search excludes papers published less than three months before the reviewed submission, preventing criticism for not citing later work.Older papers in the evaluation set are most affected by this cutoff.Source: §A.1 · p. 9
  • A context-distillation sub-agent reads selected retrieved papers in full and answers targeted queries when abstracts are insufficient or full reading would exceed the context window.Source: §A.1 · p. 9

A.2 AgenticJudge for Review Evaluation

The section describes AgenticJudge’s review-quality criteria and evaluation setup, then reports its alignment with human ratings and comparisons among reviewer systems. It also contrasts structured review with an autonomous variant.

A.2 AgenticJudge for Review Evaluation

  • Human annotations yielded five candidate review criteria; Comprehension and Substance saturated at ≥0.95 and were retained as sanity checks, while the other three form the reward.The final reward is the mean of Issue Overlap, Fabrication, and Calibration; the judge scores these through independent calls.Source: Criteria · pp. 9–10, Table 5 · p. 11
  • The evaluation uses 650 paper–review pairs from ICLR 2018–2021, selected to limit likely LLM-assisted human reviews and avoid aggressive filtering assumptions.Source: Dataset · p. 10
  • AgenticJudge achieves Spearman’s rank correlation of ρ = 0.84, raw inter-reviewer agreement of 92%, and Cohen’s κ of 0.81 against human annotations.Four experts annotated 100 randomly sampled papers using the three main discriminative metrics.Source: Human Alignment with Judge · p. 10
  • Among three reviewer backbone models, the AI Reviewer with Deepseek v4 flash aligns most with human scores and venue accept-or-reject outcomes.Figure 7 compares reviewer scores with average human ratings and counts agreement with the venue decision.Source: Score Agreement Across Backbone Models · p. 10, Figure 7 · p. 10
  • The structured reviewer consistently outperforms an autonomous variant that uses the same tools but chooses its own randomly shuffled execution strategy.The autonomous agent often skips important evaluation steps or performs them out of order.Source: Autonomous vs Structured Reviewer · p. 10, Figure 8 · p. 11
  • The reviewer is compared with fine-tuned models, LLMs, multi-agent frameworks, and the frontier LLMs used within its own system.AgenticJudge scores the systems on Overlap, Fabrication, and Calibration; Figure 6 presents results across baselines.Source: Results Across Review Baselines · pp. 10–11, Figure 6 · p. 10

A.3 Input Representation: LaTeX vs PDF

The supplement reports that PDF OCR preserves review faithfulness and calibration better than Pandoc conversion, describes how revision rounds carry forward prior work, and specifies the contents of example trajectories.

A.3 Input Representation: LaTeX vs PDF

  • Faithfulness drops to 0.365 from 0.500 and Calibration to 0.403 from 0.589 with Pandoc conversion, which breaks tables and mathematical notation.The reviewer instead uses olmOCR-2-7B-1025 to convert the compiled PDF into Markdown; Table 6 compares PDF OCR with Pandoc conversion of LaTeX.Source: §A.3 · p. 11, Table 6 · p. 11

B.1 The Revision Round

  • At round t, the scientist revises the previous version, retaining earlier drafts, reviews, and results, while the reviewer sees only the current version.The reviewer has no access to earlier drafts or scores, so its rating reflects the submitted manuscript rather than the revision sequence.Source: §B.1 · p. 11

B.2 Example Revision Trajectories

  • Table 7 reports ten randomly selected papers’ initial human-review averages, resolved weakness counts, review-score trajectories, best scores, and corresponding Stanford Reviewer score changes.Source: §B.2 · p. 11

B.3 Per-Domain Results

The appendix reports independent score trajectories, verifies persistent novelty objections, and documents the revision protocol, evaluation procedures, and computation costs.

B.3 Per-Domain Results

  • Figure 9 reports domain-level score trajectories with independent Stanford Reviewer scores, while Figure 10 shows the AI-initialized trajectory narrowing its gap with the human-initialized condition.Source: §B.3 · p. 11, Figure 9 · p. 12, Figure 10 · p. 12

B.4 Verifying Novelty Objections

  • All 23 cited papers supporting 14 verifiable persistent novelty objections were relevant to the submissions and supported the reviewers’ objections.Three of 17 persistent objections could not be verified; Table 8 covers the cited prior work.Source: §B.4 · p. 11, Table 8 · p. 13

B.5 Computation Cost

  • Each five-round revision run took about 9 hours and cost about $65 per paper; evaluating 30 papers required roughly $2,000 and 270 GPU-hours.The computational budget limited evaluation-set size; 100 papers would require about $6,500 and 900 GPU-hours.Source: §B.5 · p. 11

C.1 Reviewer Prompt

  • The reviewer prompt directs reviewers to read the paper, search prior work, assess novelty, impact, and methodology, then give evidence-based, actionable feedback.The workflow includes searches for direct competitors, methods, and baselines, plus checks of experimental design, metrics, and confounds.Source: §C.1 · pp. 12–13

C.2 Scientist and Control Prompts

  • The scientist addresses feedback through code, literature, and experiments while retaining prior work; the central contribution stays fixed, and autonomous revision uses the same prompt each round.Guidance is passed as numbered weaknesses and suggestions alongside the reviewer’s score.Source: §C.2 · pp. 13–14

C.3 Side-by-Side Judge

  • The side-by-side judge compares reviews for technical accuracy, constructive value, analytical depth, and novelty and significance, using evidence and randomized review order.External search is reserved for novelty and significance assessment, with sources constrained to the paper’s cutoff date.Source: §C.3 · pp. 14–15

C.4 Agentic Judge

  • The Agentic Judge separately checks issue overlap, fabricated facts, and review calibration against human reviews and the paper.Its calibration assessment combines decision alignment with consistency between the verdict, score, and written critique.Source: §C.4 · pp. 15–16

D Example AI Review

The example review identifies measurement and mechanism contributions alongside limitations in transfer and detection evidence. It recommends testing the attack on native pipelines to strengthen the cross-task claim.

D Example AI Review

  • The extended paper adds trigger-controlled degradation and TASR to address occlusion confounding, attributes the attack to context-routing hijack, and proposes CCS, with AUC 1.00 on its test set.The attack and cross-task transfer are also reproduced on SegGPT.Source: D Example AI Review · p. 16

Strength — self-correction

  • 22.76→15.84 clean/triggered degradation, 5.37 compromised/triggered degradation, and 76.4% total split into 30.4% occlusion plus 46.0% backdoor.Source: D Example AI Review · p. 16

Weakness — test construction for the headline claim

  • The never-poisoned-task TASR values come from tasks built on LoL images, not native SRD/SIDD pipelines, so their magnitudes are not directly comparable to native-dataset results.Source: D Example AI Review · p. 16

Weakness — detection results are a 15-query result

  • CCS, STRIP, activation clustering, and spectral signatures each report AUC or accuracy 1.00 on only 15 held-out LoL test queries.CCS is not validated on the native task suite or SegGPT.Source: D Example AI Review · p. 16

Weakness — test construction for the headline claim

  • Run TASR on the native SRD, SIDD, ADE20K, NYUv2, and FSS-1000 pipelines to test cross-task transfer under the paper’s primary evaluation protocol.Report TASR and trigger-controlled degradation alongside existing degradation values for direct comparison.Source: D Example AI Review · p. 16

D Example AI Review

  • The reviewer rates soundness, presentation, and contribution 3/4 each, gives an overall 6/10, and recommends acceptance.The review notes small numerical inconsistencies across sections.Source: D Example AI Review · p. 16
Loading 2609.14738v2…