PaperlayerView exact revision on arXiv

Source-linked AI summary

No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions

Xu Yang, Zhizhou Sha, Junbo Li, Jian Yu, Yifan Sun, Matthew Zhao, Jinrui Fang, Xinyue Guo, Yining Wu, Xu Hu, Yifu Luo, Qiang Liu, Zhangyang Wang

arXiv:2606.13044v1cs.CL

TL;DR

AI reviewers may be manipulated through visible presentation-only revisions that leave a paper’s scientific content unchanged. This paper introduces adversarial repackaging to search for such revisions and finds a 75.1% attack success rate with a mean score gain of +1.21 across three mainstream models and review templates.

  • Problem

    Evidence is limited on whether AI reviewers resist systematic score manipulation through presentation-only edits that leave methods, experiments, and results unchanged.

  • Method

    The paper introduces adversarial repackaging, an iterative closed-loop search using AI-reviewer feedback to optimize presentation-level edits while holding scientific content fixed.

  • Results

    75.1% attack success rate and +1.21 mean score gain across three mainstream models and different review templates demonstrate susceptibility to presentation-only review gaming.

  • Takeaways & Limitations

    AI-review evaluation can be systematically distorted by presentation-level manipulation, especially when revisions reframe how reviewers interpret the paper rather than merely polish its surface.

  • Takeaways & Limitations

    Presentation optimization has an effectiveness ceiling for papers with concrete experimental gaps, with gains plateauing around 5.0–5.5 and improvement concentrated in the first few rounds.

Abstract

from arXiv · show

As AI-generated reviews move from experimental tools into peer-review infrastructure, most robustness concerns have focused on explicit attacks such as hidden instructions and prompt injection. We study a harder and more policy-relevant failure mode: no hidden text, no prompt injection, and no changes to methods, experiments, figures, equations, proofs, or numerical results. The attacker modifies only presentation-level content, such as the abstract, contribution framing, related work, discussion, and narrative structure. We introduce adversarial repackaging: a closed-loop attack that uses AI-reviewer feedback to search for presentation-level revisions while keeping the scientific evidence fixed. Across three mainstream AI reviewers, adversarial repackaging achieves a 75.1% attack success rate and a mean score gain of +1.21/10. The effect is not explained by ordinary prose polishing. We also reveal that strategies that change how the reviewer interprets the paper, such as related-work repositioning and analytical discussion expansion, substantially outperform surface edits such as local polishing, table formatting, and algorithm boxes. Our analysis reveals two deeper structural failure modes. First, AI reviewers are easier to impress than to convince: highlighting strengths reliably increases perceived merit, while attempts to dissolve weaknesses frequently backfire. Second, AI reviewers can confuse the appearance of addressing a limitation with actually resolving it, allowing unchanged evidence to be reinterpreted as stronger scientific contribution. These results show that the deployment risk is not only malicious hidden instructions, but the emergence of paper presentation itself as an optimization surface. We release a contamination-free rolling benchmark and attack framework for testing whether AI reviewers remain anchored to scientific content under presentation-only edits.

B Project Website: https://xyimatvoid.github.io/ARGAR-Site/

The analysis identifies two structural failure modes in AI peer review: reviewers are easier to impress than to convince, and they may mistake the appearance of addressing limitations for actually resolving them. Consequently, unchanged evidence can be reinterpreted as a stronger scientific contribution.

  • Highlighting strengths reliably increases AI reviewers’ perceived merit, whereas attempts to dissolve weaknesses frequently backfire.
  • AI reviewers can confuse the appearance of addressing a limitation with actually resolving it.
  • Unchanged evidence can therefore be reinterpreted as a stronger scientific contribution.

1. Introduction

As AI-generated reviews enter peer-review infrastructure, the paper identifies presentation-only review gaming as a policy-relevant risk beyond prohibited hidden-text and prompt-injection attacks. It proposes resistance to such gaming as necessary for AI review automation and shows that current reviewers systematically reward presentation optimization despite unchanged scientific content.

  • Motivation: AI-generated reviews are rapidly entering peer review amid rising submission volumes and limited qualified-reviewer capacity.The paper cites deployments or trials involving AAAI 2026, ICLR 2025, and other major AI conferences.
  • Problem: Existing robustness discussions emphasize prohibited attacks such as prompt injection and hidden text, leaving systematic presentation-level manipulation insufficiently examined.The proposed risk changes presentation while preserving the paper’s scientific content.
  • Proposed condition: Resistance to presentation-only review gaming is proposed as a necessary condition for AI review automation when scientific content remains unchanged.Clearer writing may be acknowledged, but presentation optimization should not systematically inflate perceived scientific value.
  • Structural significance: The vulnerability appears consistently across multiple mainstream reviewer models and review templates, indicating a structural deficiency rather than a defect of one model.The paper warns that rewarding presentation optimization over genuine contribution improvement can distort peer-review incentives.
  • Adversarial repackaging: 75.1% attack success rate and +1.21 mean score gain are achieved by closed-loop presentation-level edits across three mainstream models and different review templates.The adversarial repackaging search holds scientific content fixed while iteratively optimizing review outcomes.

2. Related Work

Prior work on AI-review manipulation spans hidden-text attacks, surface-level perturbations, and semantic rewriting, while studies of AI-review quality document systematic limitations. However, existing attacks largely demonstrate score influence or feasibility without isolating the mechanisms by which presentation-level changes succeed.

  • AI review systems and evaluation: AI-review evaluations report score inflation, convergent focus, low agreement with human reviewers, and less perspective diversity.These studies characterize quality limitations in AI reviewing.
  • Prompt injection and hidden-text attacks: Hidden-text and prompt-injection attacks manipulate review outputs through invisible instructions but expose patchable security vulnerabilities.Such attacks are prohibited by most conferences and may lead to desk rejection when detected.
  • Surface-level textual perturbation: Surface-level NLP perturbations can inflate AI-review scores, but they are non-semantic and do not explain why the evaluation mechanism fails.Examples include synonym substitution and style transfer applied to regions reviewers attend to.
  • Paper rewriting and laundering: Abstract-only optimization uses review scores as feedback but requires over two thousand API calls per paper and relies only on scalar scores.Kaneko’s method performs multi-round search over abstract paraphrases.
  • Paper rewriting and laundering: Full-paper laundering can boost AI-review scores without violating conference policies, but unconstrained rewriting does not separate scientific content from presentation and achieves limited gains.Together with abstract optimization, these methods demonstrate attack feasibility without analyzing underlying mechanisms of success.

3. Method

The method formulates adversarial repackaging as black-box optimization over presentation-level revisions that preserve scientific content. A closed-loop system uses reviewer feedback to select and retain edits, while a two-stage evaluation criterion combines score and content-level changes.

  • Attack formulation: The attacker edits LATEX source into a revised PDF using only presentation-level changes while preserving the paper’s scientific content.The reviewer is queried as a black box without access to internal prompts or model parameters.
  • Attack formulation: The optimization maximizes favorability changes in aggregated review outcomes across numerical score and review-content dimensions.The objective compares original and modified reviews under a template comprising N independent reviews.
  • Iterative attack system: The closed-loop attack repeatedly profiles reviews, plans signal-driven strategies, edits and compiles the paper, reevaluates candidates, and updates a best-so-far version.Persistent history records reviewer signals, strategies, edit plans, and outcomes, while new candidates start from the current best version to recover from failed modifications.
  • Evaluation protocol: Candidate promotion requires a direction gate and a higher composite selection score, preventing score gains that worsen review criticism or weaken recognized strengths.The gate evaluates perceived strength, weakness severity, and net content improvement relative to the original baseline anchor.
  • Evaluation protocol: Final reporting compares attacked and original versions using Δstrength, Δseverity, Δcontent, mean score shift (ΔS), and attack success rate (ASR).ASR is the proportion of papers with ΔS≥+1.

4. Experimental Setup

The experiments use a contamination-free rolling benchmark of unpublished arXiv preprints and evaluate presentation-only attacks with three frontier AI reviewers under realistic PDF-based peer-review workflows. The main attack setting matches attacker and reviewer models, uses iterative campaigns, and compares against baselines with distinct optimization scopes.

  • Dataset: The benchmark contains only unpublished arXiv preprints and can be re-executed to incorporate newly posted submissions, with data currently through April 2026.Its construction is designed to remain contamination-free and avoid becoming stale as models evolve.
  • Reviewer models and review generation: Three frontier reviewers—Claude Sonnet 4, Claude Sonnet 4.5, and GPT-5-mini—each generate N independent reviews to reduce reviewer stochasticity.Reviews use compiled PDFs rather than plain text and the complete official ICLR, NeurIPS, and ICML reviewer guidelines.
  • Attack configuration: The main experiments use matched attacker and reviewer models across multiple rounds of the six-stage attack loop, while a separate pairwise judge avoids shared biases.Cross-model transferability is analyzed separately in Appendix F.
  • Baselines: The baselines include Zero-shot Paper Laundering, which performs one full-paper rewrite after one review round, and PAA, which iteratively optimizes scores by modifying only the abstract.Their primary differences from the proposed system are, respectively, the absence of closed-loop iterative optimization and the restriction to abstract edits.

5. Results and Analysis

Presentation-only adversarial repackaging systematically improves AI-review scores without changing scientific evidence. The gains arise because reviewers respond more strongly to amplified strengths and interpret reframing as limitation resolution than to surface-level polishing.

  • Overall effectiveness: +1.21/10 mean score gain and 75.1% attack success rate result from presentation-only edits across all three AI reviewers, with statistically significant full-dataset score changes.Methods, experiments, figures, and numerical results remain unchanged; baseline scores generally move from rejection toward borderline acceptance.
  • Strength–weakness asymmetry: 86.1% of rounds improve perceived strengths, whereas 31.6% backfire by increasing weakness severity, making reviewers easier to impress than to convince.Strength gains average +2.19, and weakness backfires occur 2.6 times as often as strength backfires.
  • Strength–weakness asymmetry: 15.8% of score-improving rounds worsen weaknesses simultaneously, showing that salient new strengths can swamp harsher criticism.Across all rounds, 67.7% enhance strengths while alleviating weaknesses, but 18.4% enhance strengths while worsening weaknesses.
  • Strength–weakness asymmetry: 79.2% of papers have greater mean strength gains than mean weakness reductions, while 79.9% of failed rounds still enhance strengths and 45.3% worsen weaknesses.These patterns indicate that attack failures primarily reflect weaknesses resistant to dissolution rather than insufficient strength gains.
  • Strategy effectiveness: 87.2% of first successful rounds include contribution-list enhancement, followed by analytical discussion expansion at 66.0%, related-work repositioning at 44.7%, and abstract reframing at 42.6%.These strategies alter perceived significance and positioning without changing experimental data or methodology.
  • Strategy effectiveness: 49.3% accepted exposure for related-work repositioning and 44.9% for analytical discussion expansion exceed surface edits: table formatting 29.8%, local polishing 27.8%, and algorithm boxes 26.5%.The overall accepted-exposure baseline is 30.8%, indicating that reframing strategies outperform ordinary prose polishing.
  • Failure modes: Limitation laundering removes or softens all six original weaknesses without resolving scientific content, because reviewers confuse the appearance of addressing an issue with actual resolution.Three weaknesses disappear, one becomes a new strength, and three are softened through mechanisms such as reframing scarcity as design intent.

6. Conclusion

The paper proposes resistance to presentation-only review gaming as necessary for AI-review automation, but finds current reviewers fail when scientific content is fixed and presentation alone changes scores. It releases an attack framework and contamination-free dataset for reusable adversarial evaluation.

  • Conclusion: Presentation-only edits raise review scores across multiple mainstream models and review templates while scientific content remains entirely fixed.This demonstrates that current AI reviewers fail the proposed resistance condition.
  • Conclusion: Reframing how reviewers understand a paper is more effective than improving its surface appearance.The conclusion identifies presentation-level interpretation as the structural source of susceptibility.
  • Conclusion: The paper releases its attack framework and contamination-free dataset as a reusable benchmark for adversarial evaluation of AI review systems.The authors position these resources for systematic testing of review-system robustness.

Limitations

The experiments cover three reviewer configurations from Claude and GPT families but not additional models due to compute constraints. Presentation optimization has a bounded effectiveness ceiling on papers with concrete experimental gaps, with scores plateauing around 5.0–5.5 and improvements concentrated early.

  • Model coverage: Three reviewer configurations span Claude and GPT series, but additional models were not tested because of compute constraints.Extending evaluation to further model families would strengthen the conclusion.
  • Model coverage: Cross-model transfer results suggest the vulnerability reflects a structural property of current AI reviewers rather than a model-specific artifact.Attacks optimized against one model remained effective on another.
  • Effectiveness ceiling: 5.0–5.5 is the approximate score plateau for papers whose weaknesses involve concrete experimental gaps, despite continued presentation optimization.Examples include single-dataset evaluation and absent real-world validation.
  • Effectiveness ceiling: Most score improvement occurs in the first few rounds, indicating that presentation-based gains are bounded when substantive shortcomings remain.AI reviewers retain partial sensitivity to concrete experimental weaknesses.

Ethical Considerations … C. System Architecture Details

The paper frames its presentation-only attack as a dual-use vulnerability and releases a reproducible, contamination-free robustness-testing pipeline rather than venue-targeted attack tools. Its appendices detail strategy mechanisms, dataset construction, and the three-layer system architecture supporting the edit–compile–review workflow.

  • Ethical Considerations: Presentation-level edits can systematically manipulate current AI reviewers, motivating disclosure despite the findings’ dual-use nature.The edits remain within normal academic writing practices and require no specialized tools.
  • Ethical Considerations: The released pipeline supports reproducible robustness testing without targeting specific venues or platforms, using only public arXiv preprints and no private or personally identifiable data.Users can construct their own contamination-free evaluation datasets through the pipeline.
  • B Dataset Construction: The benchmark collects arXiv preprints with compiled PDFs and editable LATEX sources across machine-learning, computer-vision, and natural-language-processing categories, using an automated retrieval pipeline.The collection can be re-executed to incorporate newly posted preprints.
  • B.1. Construction pipeline: The filtering pipeline deduplicates versions, enforces 12–35-page and at-least-two-author criteria, verifies publication status using independent sources, and discards incomplete or uncompilable sources.Dual verification has retention ≈67%, while source compilation verification has retention ≈29%.
  • A. Presentation-Level Strategy Pool: Narrative restructuring changes how reviewers interpret contributions, positioning, and limitations, whereas surface editing improves presentation quality without changing narrative framing.The strategy pool categorizes these mechanisms according to the observed strategy effectiveness gradient.
  • B.1. Construction pipeline: The strategy pool includes contribution-list strengthening, analytical discussion expansion, related-work repositioning, abstract reframing, introduction restructuring, and conclusion restructuring.These strategies respectively emphasize strengths, analytical completeness, novelty positioning, concrete results, motivation, and assertive restatement of contributions.
  • B.1. Construction pipeline: Additional strategies include narrative repositioning, theoretical formalization, impact exposition, full rewriting, weakness removal, formatting cleanup, prose refinement, table packaging, algorithm boxes, section renaming, and title repositioning.The passages distinguish narrative and formalization changes from local polishing, formatting, and organizational edits.
  • C. System Architecture Details: The retained dataset has median 18 pages and 4 authors, spans 11 research areas, and uses multi-model cross-review under official ICLR, NeurIPS, and ICML guidelines; Figure 4 details the three-layer architecture and attack-round information flow.The most represented areas are Deep Learning (33%) and ML for Sciences (17%).

D. Experimental Details

The experiments use three multimodal API reviewer checkpoints with controlled temperatures and structured review generation. Attack campaigns run in matched-model settings with adaptive direction-gated optimization, while baselines isolate zero-shot rewriting and fixed-strategy approaches.

  • Reviewer models: Three reviewer checkpoints are used: claude-sonnet-4-2025, claude-sonnet-4-5-20250929, and gpt-5-mini.All reviewers receive compiled PDFs through multimodal APIs.
  • Reviewer models: Claude reviewer temperature is 0.1, while GPT-5-mini uses its API-mandated default temperature of 1.0.The Claude setting supports reproducibility while preserving minor stochastic variation; GPT-5-mini does not support user-configurable temperature.
  • Review generation: Each review prompt combines a system instruction, complete venue guidelines, and the paper PDF, producing JSON reviews with scores and strengths, weaknesses, and questions.ICLR 2025 guidelines are the default, with NeurIPS and ICML substituted in cross-template transferability experiments; each round generates N=3 independent reviews and aggregates their mean score.
  • Attack agent configuration: Matched attack agents use the target reviewer checkpoint, run for up to 8 rounds, and apply direction-gate thresholds τ_s=1.0, τ_w=1.0, and τ_n=0.8.Candidates must improve perceived strengths by at least 1.0, avoid worsening weakness severity by more than 1.0, and achieve net benefit above 0.8; selection weights are w_r=0.8 and w_c=0.2.
  • Baseline implementation: The reproduced Zero-shot Paper Laundering baseline performs one full-paper rewrite after one review round, while PAA uses fixed high-frequency strategies without adaptive adversarial optimization.PAA uses the same number of reviewer queries as the attack system but omits an adversarial objective, signal-driven strategy selection, and a direction gate.

E. Robustness Validation of LLM-Based Evaluation

The evaluation pipeline is validated along four complementary dimensions: alignment with historical peer review, test-retest reliability, directional content calibration, and order invariance. These validations support the reliability of both AI reviewer scores and pairwise judge comparisons.

  • Score calibration: AI reviewer scores are significantly higher for accepted papers and correlate moderately with historical human mean scores, with Pearson r=.58–.61 and Spearman ρ=.56–.58.All three models show p<.005 for accepted-versus-rejected score differences; calibration weakens with more mid-scoring papers but does not affect directional score-change analyses.
  • Test-retest reliability: +1.21 is the cross-model mean attack score improvement, far exceeding the small within-paper scoring variance observed across N=3 independent reviews.The within-paper standard deviation is small for all three models on the ICLR 1–10 rating scale.
  • Content calibration: Pairwise-judge directional accuracy improves monotonically as human review-rating gaps increase across nested ICLR 2024 and 2025 samples.The nested design contains 64 pairs with gap ≥5, 80 with gap ≥4, and 96 with gap ≥3.
  • Order invariance: On 96 review pairs with gap ≥3, directional consistency is 100% for Δseverity and ≥98% for Δstrength after reversing input order.These results indicate robustness to input order, especially for criticism severity.
  • Summary: Together, the four validations show that reviewer scores track historical outcomes, attack effects exceed natural variance, and pairwise judgments capture content differences consistently.The validations cover score calibration, test-retest reliability, content calibration, and order invariance.

F. Transferability · G. Human evaluation · G.1. Blind pairwise semantic-preservation audit.

Presentation-level attacks transfer across reviewer models and review templates, while blind human audits find that the attacked manuscripts largely preserve scientific content. The main semantic differences concentrate in narrative framing, especially conclusions and main claims.

  • F. Transferability: The transfer experiments reuse attacked papers and only re-evaluate them with different reviewer models or review templates, without rerunning the attack.This tests whether presentation changes depend on the original reviewer model or ICLR-specific criteria.
  • F. Transferability: +0.88 mean ΔS in mismatched settings, compared with +1.42 in matched settings, while every off-diagonal cross-model transfer result remains positive.Transfer within the Sonnet family is stronger than cross-family transfer, but the weakest cross-family pair remains positive.
  • F. Transferability: ICLR-optimized attacks produce positive score gains under both NeurIPS and ICML review guidelines, despite differing rating scales.Absolute ΔS values are not directly comparable across conference scales, but the effect is positive within each scale.
  • G.1. Blind pairwise semantic-preservation audit.: Annotators compared randomized Version A and Version B manuscripts without knowing which was original or modified, using five scientific-content dimensions scored from 0 to 2.The dimensions were contribution, method, experimental setup, reported evidence, and conclusions or main claims.
  • G.1. Blind pairwise semantic-preservation audit.: The audit classified a pair as preserved when its aggregated score reached τ= 0.8, corresponding to at least 8 out of 10 across annotators and dimensions.Annotator-level scores summed the five dimension ratings and normalized by the maximum score of 10 before aggregation.
  • G. Human evaluation: Three human annotators independently evaluated 30 blinded version pairs, yielding a mean preservation score of 0.80, 20/30 preserved pairs, and a 66.7% preservation rate.Agreement was 83.3% by percent agreement, with Fleiss’ κ= 0.62.
  • G.1. Blind pairwise semantic-preservation audit.: Experimental setup had the highest dimension-level preservation score at 1.93, whereas conclusions and main claims had the lowest at 0.97.Method and experimental setup were near-ceiling, while lower scores concentrated on presentation-oriented narrative framing.

H. Case Study: Full Analysis … J. Prompt Templates

The case study shows that presentation-only edits can shift an AI review from rejection to acceptance without changing scientific evidence, by inflating strengths and laundering unresolved limitations. Discussion findings rule out noise and single-model explanations, identify deployment risks, and motivate content-anchored robustness testing using shared review prompts and cross-template evaluations.

  • H. Case Study: Full Analysis: 3/10 to 6/10: presentation-only edits moved the case-study review from Reject to Weak Accept without changing methods, experiments, figures, or numerical results.The original review listed 6 substantive weaknesses, while the post-attack review upgraded all three sub-scores.
  • H. Case Study: Full Analysis: The reviewer echoed self-positioning language, treating “complementary data regimes,” “the first,” explanations, propositions, and acknowledged limitations as evidence of stronger contributions.This reflects confusion between appearing to address an issue and actually resolving it.
  • H. Case Study: Full Analysis: 3 of 6 weaknesses disappeared and the rest softened, although none of the criticized scientific issues was resolved.The attack used four mechanisms: methodological rationalization, scarcity reframing, decoy limitations, and theoretical formalization.
  • I.1. Alternative Explanations: +1.21: the attack’s mean score gain exceeded the reviewer’s within-paper standard deviation of 0.10–0.28, while calibration and order-invariance checks supported robustness of the measured effect.The pairwise judge achieved directional accuracy >90% at larger gap thresholds, and its judgments remained ≥98% consistent when input order was swapped.
  • I.1. Alternative Explanations: Presentation-only attack effects appeared across Sonnet 4, Sonnet 4.5, and GPT-5-mini, and transferred across mismatched models and ICLR, NeurIPS, and ICML templates.These results argue against the attack working only on one reviewer model or template.
  • I.2. Broader Significance: Presentation-only manipulation is harder to defend against than prompt injection because edits are visible, legitimate, and indistinguishable from ordinary clarification without judging their fit to the underlying science.The same related-work repositioning or contribution framing can clarify genuine contributions or inflate novelty.
  • I.2. Broader Significance: 75.1% ASR and +1.21 score gain can make strategic packaging more rewarding than better science, even without malicious intent.The paper characterizes this as an incentive-structure distortion caused by low attack cost and substantial payoff.
  • I.3. Implications for Deployment: Deployment should test score stability under presentation-only perturbations, benchmark content anchoring, and separate scientific soundness judgments from writing-quality judgments.The released attack framework and contamination-free dataset support such testing before AI reviewers influence acceptance decisions.
Loading 2606.13044v1…