How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review

Method

• The study constructs a controlled corpus of 4,200 full-paper manuscripts from 120 anonymized ICLR 2026 submissions, varying six rhetorical dimensions in opposing directions.Matched rewrites preserve underlying scientific content and key methodology elements while allowing full-narrative changes, including captions, transitions, and reported-result presentation.Source: §1 · pp. 2–3

Findings

• Evidence framing and novelty stance produce the largest rhetorical effects, followed by scope framing, with effects depending on the reviewer’s initial score.Positive evidence framing raises Overall Assessments by up to 0.93, negative novelty stance lowers them by up to 0.73, and evidence framing changes weak-accept probability by 13 percentage points on average; lower initial scores tend to rise, higher scores tend to fall, and contrasts are strongest in middle ranges.Source: Figure 2 · p. 3, §1 · p. 3

Method

• The framework combines controlled source-level rewriting, multi-model AI review, paired comparisons, and single-dimension, joint, recursive, and reviewer-guided interventions.Reviews use standard and strict protocols, with structural checks, compilation, and repair supporting reviewable full-paper variants.Source: §1 · pp. 3–4

Findings

• More elaborate rewriting produces diminishing, configuration-dependent gains rather than reliably improving scores.Joint rewriting yields substantial gains with Opus 4.8 but near-zero gains with GPT-5.5; reviewer guidance does not consistently outperform an unguided second pass, and gains diminish after the second pass.Source: §1 · p. 3
• Rewriters primarily determine separation between opposing variants, whereas reviewers determine the magnitude and sign of rewrite-induced Overall Assessment effects.The reviewer also determines whether Overall Assessment changes appear mainly in contribution or soundness.Source: §1 · p. 3
• Strict review lowers absolute Overall Assessment scores without consistently strengthening or weakening rhetorical effects.The review protocol shifts the score scale, but does not reliably change rewrite sensitivity.Source: §1 · p. 3

2 Related Work

• Prior studies find that rubric-based LLM evaluations are sensitive to prompts, presentation, sampling, and evaluator identity, while scientific-review models inconsistently identify weaknesses and paper quality.These findings span broader preference evaluations and scientific peer review.Source: §2 · p. 4
• Manuscript-side interventions—including hidden instructions, artificial perturbations, and meaning-preserving rewrites—can alter AI-review outcomes, but prior rewriting work rarely characterizes rhetorical effects systematically.The paper positions its analysis as addressing this gap.Source: §2 · p. 4

3 A Controlled Analysis Framework for Rhetorical Rewriting in AI Review

• The overall framework combines verified content-preserving variants, blinded review, and within-paper contrasts across single, joint, recursive, and reviewer-guided rewriting.This design measures whether rhetorical presentation changes judgments while keeping the underlying scientific contribution and evaluation condition controlled.Source: §3 · p. 4, Table 2 · p. 5

3.1.1 Seed Corpus and Baseline Construction

• The study samples 120 ICLR 2026 submissions across six human-rating intervals and verifies an anonymized manuscript baseline for independent variant generation.The corpus contains 20 papers from each interval, while source and PDF checks confirm removal of identifying information without changing other content.Source: §3.1.1 · p. 5

3.1.2 Rhetorical Intervention Design

• Six prespecified rhetorical dimensions are rewritten in opposing directions from the same baseline to separate directional sensitivity from rewriting-induced score movement.The dimensions cover novelty, scope, evidence framing, contribution structure, technical register, and lexical or syntactic complexity.Source: §3.1.2 · p. 5

3.1.3 Source-Level Rewrite Harness

• The source-level harness applies interventions across complete LaTeX projects while preserving methods, settings, reported values, evidence boundaries, findings, and protected structural anchors.The reported overview identifies evidence framing and novelty stance as producing the largest assessment changes, followed by scope framing.Source: §3.1.3 · pp. 5–6, Table 3 · p. 6

3.2 Single-Dimension Rewriting

• Each paper receives 12 independently generated single-dimension variants—both directions for all six dimensions—from the same anonymized baseline.No single-dimension intervention is layered on top of another, producing 2,880 rewritten manuscripts in total.Source: §3.2 · p. 6

3.3 Enhanced Rewriting

• Enhanced workflows test simultaneous six-dimension rewriting, three-round recursion, and reviewer-guided second-pass revision under shared preservation requirements.Joint rewriting produces 240 manuscripts, recursive rewriting produces 480, and reviewer-guided rewriting produces 480 Intermediate and Final rewrites.Source: §3.3 · pp. 6–7

3.4 AI Review Design and Outcomes

• Five blinded AI reviewers evaluate compiled originals and rewrites under standard or strict prompts, using overall rating as the primary outcome.Secondary outcomes are soundness, presentation, and contribution; weak-accept probability is defined as the share of ratings at least 6.Source: §3.4 · p. 8

3.5 Paired Analysis

• Paired comparisons hold paper, reviewer model, rewrite model, and protocol fixed to estimate score movement and directional separation.Analyses report signed and absolute changes with 95% percentile intervals from 5,000 paper-level bootstrap resamples, separately for standard and strict protocols.Source: §3.5 · p. 9

4.1 Evidence Framing and Novelty Stance Most Readily Shape AI-Review Outcomes

• Evidence framing and novelty stance produce the largest, most consistent OA contrasts, with scope framing weaker and other dimensions smaller or less stable.Positive evidence and novelty variants are favored across configurations; weak-accept contrasts are 13.0, 12.0, and 9.0 percentage points for evidence, novelty, and scope, respectively.Source: §4.1 · p. 9

4 Which Rhetorical Dimensions Matter Most?

• Across conditions, rhetorical sensitivity is selective rather than a general reward for stronger rhetoric, with evidence framing moving scores in both directions and novelty and scope effects mainly penalty-driven.Secondary scores show larger novelty and evidence contrasts in contribution and soundness than presentation, while unfavorable scope mainly lowers contribution.Source: §4.1 · p. 9, §4 · p. 9

4.2 The Dimension Hierarchy Is Broadly Preserved across Human-Assessed Quality Levels

• The dimensional hierarchy persists across human-assessed quality levels, without rewrite effects consistently increasing or decreasing with human-rated quality.Evidence, novelty, and scope retain the clearest contrasts, each showing the intended ordering for more than 100 of 120 manuscripts; strict prompts preserve the pattern.Source: §4.2 · p. 9

4.3 The AI Reviewer’s Starting Score Strongly Conditions the Direction of Change

• The AI reviewer’s starting score strongly conditions movement: low initial scores tend to rise, whereas high initial scores tend to fall.Under standard evaluation, positive evidence variants change OA by +1.05 for original scores in [1, 3] but −0.19 for scores in [8, 10]; effects may reflect scale bounds and regression to the mean.Source: §4.3 · p. 10

5.1 Joint Rewriting Produces Rewriter-Dependent Gains

• +0.289 standard and +0.463 strict reviewer-averaged gains occurred for Opus 4.8 joint rewrites, versus +0.021 and +0.045 for GPT-5.5.Only Opus 4.8 produced positive mean changes for every reviewer under both prompts; GPT-5.5’s near-zero means reflected heterogeneous reviewer responses.Source: §5.1 · p. 11
• +1.42 mean OA change for initial scores in [1, 3] declined to −0.88 for scores in [8, 10], while effects remained comparatively stable across populated human OA ranges.This descriptive pattern tracked the AI reviewer’s starting score more closely than human-assessed paper quality and may partly reflect scale bounds and regression to the mean.Source: §5.1 · pp. 11–12

5.2 Reviewer Guidance Can Improve on One Pass but Adds Little beyond Repetition

• +0.204 to +0.250 overall increase was driven by Opus 4.8, while guided rewriting underperformed an unguided second pass in all four rewriter-prompt averages.Guided-minus-unguided contrasts were −0.035 and −0.042 for GPT-5.5, and −0.008 and −0.067 for Opus 4.8; the comparison is descriptive because first-pass rewrites were independently generated.Source: §5.2 · p. 12

5.3 Additional Rewrite Rounds Produce Uneven Gains across Configurations

• +0.410 standard and +0.624 strict gains by round two for Opus 4.8 rose only to +0.080 and +0.153 by round three for GPT-5.5.The third round added little for Opus 4.8, while reviewer-specific trajectories sometimes flattened or partially reversed earlier gains.Source: §5.3 · p. 12, Figure 3 · p. 12

5 Do More Complex Rewrites Yield Reliable Gains?

• More elaborate rewriting yields configuration-dependent and diminishing gains: joint-rewrite benefits are substantial with Opus 4.8 but near zero with GPT-5.5, while later rounds add little.Across multi-stage conditions, score gains depend on the rewriter-reviewer configuration and generally concentrate by the second round.Source: §5 · p. 11, §5.3 · p. 13

6.1 Reviewer and Rewriter Models Produce Different Patterns of Rewrite Effects

• Reviewers share preferences for positive novelty, evidence, and scope variants but differ in score magnitude, direction, and overall level, whereas rewriters mainly control rhetorical separation.Qwen 3.5 F and GPT-5 mini show the widest response distributions, while GPT-5.5 moves less and Sonnet 5 shifts effects downward; Opus 4.8 generally yields wider separation than GPT-5.5.Source: §6.1 · p. 13, §6 · p. 13

6.2 Strict Review Lowers Scores but Does Not Consistently Change Rewrite Effects

• Strict review lowers mean OA from 6.296 to 4.934, yet rewrite effects change only from +.003 to +.020 and remain weakly aligned across protocols.The standard–strict paper-level correlation is .862 for absolute OA but only .081 for rewrite effects, indicating recalibrated severity without greater invariance to rhetorical variation.Source: §6.2 · p. 14, Table 8 · p. 14

6.3 Associations with Secondary Scores Vary across Reviewers

• Reviewers map similar OA changes onto rubric dimensions differently, with contribution and soundness generally more associated than presentation, limiting rubric-level interpretability.Qwen 3.5 F aligns OA changes mainly with contribution, whereas strict Gemini 3.5 FL emphasizes contribution and soundness; these are co-movements, not evidence of causal mediation or changed merit.Source: §6.3 · p. 14, Table 9 · p. 15
• Strict protocols lower mean OA by 1.36 points without consistently strengthening or weakening rewrite sensitivity.Reviewer-specific changes differ: Qwen 3.5 F becomes less positive, GPT-5 mini more positive, and Sonnet 5 remains negative under strict review.Source: §6.3 · p. 14, §6.2 · p. 14

Conclusion

• Rhetorical sensitivity varies by dimension, rewriting process, rewriter and reviewer models, and review protocol, rather than reflecting one stable average effect.Accordingly, AI-assisted review should be evaluated for rhetorical robustness across multiple models and conditions.Source: §7 · p. 15

• The benchmark covers only ICLR 2026 submissions with recoverable full-paper sources and public review metadata, limiting generalization across venues and scientific fields.Source: Limitations · p. 15
• Computational expense caused some planned reviews to lack valid structured records, with failures potentially related to paper topics triggering model safeguards.Source: Limitations · p. 15
• Because the six rhetorical dimensions are nonorthogonal and rewritten as full-paper interventions, effects cannot be interpreted as independent coefficients for individual linguistic features.Source: Limitations · p. 15
• Most configurations use one AI review per manuscript, so findings characterize the tested models and prompts rather than human review or all variation from repeated model sampling.Repeated reviews were examined only in an auxiliary audit.Source: Limitations · p. 15

Ethical Considerations

• The study uses publicly available review metadata and public arXiv sources.Source: Ethical Considerations · p. 15
• Before API submission, manuscripts were stripped of names, affiliations, acknowledgments, status statements, and identity-bearing links.Source: Ethical Considerations · p. 15
• Results are reported in aggregate, and individual authors are not assessed.Source: Ethical Considerations · p. 15
• The framework is dual use: it can diagnose rhetorical sensitivity but also tailor manuscripts to AI-reviewer preferences.Source: Ethical Considerations · p. 15
• Score increases should not be interpreted as improvements in scientific merit or used to manipulate active review processes.Source: Ethical Considerations · p. 15
• Such optimization could reward rhetorical assertiveness over evidential improvement and distort scientific evaluation.Source: Ethical Considerations · p. 15

A.1 LLM-Based Evaluation

• Prior work shows that LLM judgments vary with presentation, prompt design, evaluator identity, and evaluation conditions, motivating systematic tests of content-preserving rhetorical interventions.Related studies report order, length, wording, anchors, familiarity, sampling, model-family, verifiability, and adaptive-style effects, while scientific-review systems remain uneven in detecting substantive weaknesses.Source: §A.1 · p. 19, §A.2 · pp. 19–20

A.3 Rhetorical Rewriting and Presentation Effects

• The study differs from score-seeking or adversarial rewriting by applying six prespecified rhetorical dimensions in opposing directions from the same manuscript baseline and retaining lower-scoring variants.The design measures effects on overall ratings, scientific subscores, weak-accept decisions, score ranges, rewriting models, reviewer models, and protocols.Source: §A.3 · p. 20

B.1 AI-Review Execution

• The evaluation infrastructure used provider APIs and PDF inputs without OCR, browsing, retrieval, metadata about rhetorical conditions, or overridden sampling controls.These execution choices define the implementation conditions for the AI-review experiments.Source: §B.1 · p. 21

B.2 Review Missingness and Potential Safeguard Triggers

• 42,396 of 42,480 planned reviews yielded valid structured records, with 84 evaluations retained as missing and Sonnet 5 failures concentrated in 26 JULI-related cells under possible safety-triggering conditions.The records do not establish that safeguards caused every failure, and one Qwen strict joint-rewrite evaluation was also rejected by provider inspection.Source: §B.2 · p. 21

B.4 Comparison with Human Overall Assessments

• Under the standard prompt, all five AI reviewers scored above human OA by 0.418–2.934 points, while strict prompting lowered three reviewers below human OA, nearly aligned Qwen 3.5 F, and left Gemini 3.5 FL 1.676 points higher.Across reviewers and prompts, paper-level MAE ranged from 1.029 to 2.934 points and Spearman’s ρ from 0.395 to 0.621.Source: §B.4 · p. 22, Table 11 · p. 22

C Detailed Results for Individual Rhetorical Dimensions

• The appendix extends Section 4 with pooled and fixed-model overall-rating effects, weak-acceptance results and directional crossings, and score-range heterogeneity, while preserving paper-level and fixed-pair comparisons.Table 12 reports pooled means and paired changes, Table 13 fixes both model axes, and Tables 18–19 report fixed-pair weak-accept rates and upward and downward crossings.Source: C Detailed Results for Individual… · p. 23, §C.1 · p. 23, §C.5.2 · p. 27

C.2 Fixed-Model Directional Heterogeneity

• Evidence, novelty, and scope produce the most consistent directional effects, whereas technical register, contribution structure, and lexical complexity are smaller or less stable across fixed reviewer–rewriter pairings.This hierarchy is supported by both aggregate fixed-model comparisons and paper-level consistency analyses.Source: §C.2 · p. 24, §C.4 · p. 25

C.3 Secondary AI-Review Score Spillovers

• Secondary score changes in presentation, contribution, and soundness are interpreted as reviewer-response spillovers because rewrites preserve methods and reported values, while paper-level directional effects are distributed across the corpus.Novelty, evidence, and scope show the intended ordering for more than 100 of 120 papers; lexical complexity is nearly balanced at 62 wins, 7 ties, and 51 losses.Source: §C.3 · p. 24, Table 14 · p. 24, §C.4 · p. 25

C.7 AI Original-Score Range Analysis

• The four AI original-score ranges are assigned using prompt- and reviewer-matched original overall ratings, so one paper may appear in different ranges across reviewers.Unique-paper counts across ranges are therefore not mutually exclusive.Source: §C.7 · p. 28
• Extreme-range profiles should not be treated as unbiased estimates of baseline-quality moderation because bounded scales, regression to the mean, and sparse reviewer-specific cells can influence movement.The ranges are retained as descriptive diagnostics rather than independent subgroup claims.Source: Table 21 · p. 29, §C.7 · p. 30

C.7.1 Panel-Aligned Values for the Main-Text Displays

• Panel-aligned tables report paper-level mean rating changes and weak-accept probability changes across original-score ranges, averaging all five reviewers within each paper.The two rewrite models remain separate, and weak-accept changes are reported in percentage points.Source: §C.7.1 · p. 30, Table 23 · p. 30, Table 24 · p. 31

C.7.2 Pooled Overall-Rating Inference

• Pooled inference summarizes original means, rewrite means, signed changes, absolute movement, and paper-bootstrap intervals across ranges, rhetorical dimensions, directions, and protocols.Both rewrite models and five reviewer models are averaged within paper.Source: §C.7.2 · p. 31, Table 25 · p. 32

C.7.3 Pooled Rating-6 Transitions Near the Decision Boundary

• Near the rating-6 boundary, pooled transition counts distinguish upward-only movement from [4,5] and downward-only movement from [6,7].Full transition-count decompositions are omitted for extreme ranges because they are farther from the boundary and some fixed cells are sparse.Source: §C.7.3 · p. 31, Table 26 · p. 34

C.7.4 Full Reviewer-Model Decomposition

• The full decomposition keeps rewrite and reviewer models fixed so reviewer-specific baselines and responses remain available after pooled and boundary summaries.Each cell reports the original value, rewrite value, and change.Source: §C.7.4 · p. 36, Table 27 · p. 37

C.7.5 Detailed Secondary-Score Profiles Across the Rating Scale

• Secondary-score profiles combine presentation, contribution, and soundness responses across original overall-rating ranges as secondary diagnostics rather than primary range-specific effects.The ranges are defined by original overall rating, not by each secondary score itself.Source: §C.7.5 · p. 36, Table 29 · p. 45

D Detailed Results for Multi-Stage Rhetorical Rewriting

• The appendix distinguishes joint rewriting, recursive reapplication, and reviewer-guided comparisons, including an independently generated no-review endpoint that is not a matched causal treatment.The reviewer-guided comparison uses the prompt-matched original and Final manuscript, whereas its no-review reference comes from a separate two-pass pipeline.Source: D Detailed Results for Multi-Stage… · p. 47

D.1 Joint Rewrite by Reviewer Model

• Joint-rewrite analyses fix the rewriter, review prompt, and reviewer while comparing matched endpoint effects with paper-matched means of six positive single-dimension rewrites.The cross-batch joint-versus-single-dimension comparison is descriptive rather than a formal interaction or synergy estimate, and score ranges are reported separately by reviewer and prompt.Source: §D.1 · p. 47, Table 30 · p. 47, Table 31 · p. 48

D.2 Final Reviewer-Guided Endpoints by Reviewer Model

• Reviewer-guided endpoint analyses report original and Final levels, paired changes, threshold shifts, and intervals across all 20 reviewer configurations.They also compare Final manuscripts with independently generated no-review two-pass endpoints using a descriptive contrast.Source: §D.2 · p. 48, Table 32 · p. 48

D.3 Complete Recursive Joint-Rewrite Trajectories

• Recursive joint-rewrite analyses track cumulative overall-rating and weak-accept-probability changes through Round 3, both pooled across reviewers and decomposed by reviewer model.The trajectories are normalized to matched originals, with missing evaluations averaged over available reviewers.Source: §D.3 · p. 49, Table 33 · p. 49, Table 34 · p. 49

E Detailed Reviewer and Secondary-Score Results

• The reviewer analyses retain the complete prompt-level calibration comparison while adding reviewer-specific differences and cross-reviewer consistency analyses.The additional results focus on how reviewers differ and how consistently they respond across evaluations.Source: E Detailed Reviewer and… · p. 51

E.1 Five-Reviewer Comparison

• Reviewer models preserve broad paper ordering but differ in OA changes after rewriting across the 12 single-dimension rewrites.Figure 4 gives full response distributions for five reviewers, while Table 37 reports exact mean changes and pairwise rank-correlation ranges.Source: §E.1 · p. 51, Table 37 · p. 51

Secondary-score associations

• Paper-first associations varied by reviewer and prompt, with contribution–OA correlations reaching .791 and soundness–OA correlations reaching .587, while presentation associations were often near zero.The analysis averaged the 12 single-dimension rewrites within each paper and used paper-bootstrap intervals.Source: §E.2 · pp. 51–52, Table 38 · p. 51

Cross-reviewer consistency

• Near-zero cross-reviewer agreement showed that reviewers generally disagreed on which papers exhibited the largest rewrite responses across OA and all secondary scores.This disagreement persisted even for reviewer pairs with similarly signed aggregate changes.Source: §E.2 · p. 53, Table 39 · p. 53

Secondary-score associations

• .020 to .239 presentation correlations contrasted with .210 to .691 for soundness and .171 to .841 for contribution after removing each paper’s mean response.Across 20 rewriter–reviewer–prompt combinations, soundness and contribution changes therefore co-moved with OA more strongly than presentation changes.Source: §E.2 · p. 53

F Stability Under Repeated Review Sampling

• Pearson r = 0.995 and Spearman ρ = 0.993 for 12 baseline-paired effects, while positive–negative contrasts reach Pearson r = 0.996 and differ by at most 0.044 points when averaging three reviews.Single-review and three-review-mean estimates remain nearly unchanged, supporting one designated review per evaluation cell in the main analyses.Source: F Stability Under Repeated Review… · p. 54, Table 40 · p. 54

G Complete Rewrite and Review Prompt Templates

• The complete rewrite templates combine one dimension-and-direction instruction with shared full-paper requirements, while reviewer feedback is inserted only into the reviewer-guided Final rewrite.Review prompts retain complete evaluation and structured-output instructions for each PDF, and joint prompts are reused across rounds.Source: G Complete Rewrite and Review Prompt… · p. 55

G.1 Direction-Specific Rewrite Prompts

• Direction-specific prompts alter only the targeted rhetorical property while preserving the manuscript’s underlying scientific content, including claims, scope, quantitative facts, contributions, technical content, and reported results.The six dimensions are claim and novelty stance, scope and generalization, quantitative evidence framing, contribution structure and salience, technical register and formalism, and lexical and syntactic complexity.Source: §G.1 · pp. 55–56

G.2 Shared Full-Paper Rewrite Requirements

• Shared requirements mandate systematic paragraph-level rewriting across every major paper section, with visible target-dimension edits while preserving the paper’s scientific content.Rewriters may reorganize sentence structure, combine or split sentences, change paragraph framing, and revise transitions.Source: §G.2 · p. 57
• Full-paper rewrites must edit prose broadly while preserving scientific claims, methods, data, comparisons, findings, protected structures, and numerical meaning.Rewriters may alter wording and framing but may not add experiments, data, baselines, numbers, unsupported conclusions, or invalid protected LaTeX and file elements.Source: §G.2 · p. 57

G.3 Joint and Reviewer-Guided Rewrite Prompts

• Joint rewriting coordinates six editorial dimensions holistically, while reviewer-guided rewriting reconciles that profile with validated feedback without exceeding the manuscript’s scientific and evidential boundaries.The dimensions target claims and novelty, scope, empirical interpretation, contribution salience, technical precision, and lexical-syntactic sophistication; reviewer feedback can moderate excess or imbalance.Source: §G.3 · pp. 58–59

G.4 Review Prompts

• Standard and strict reviews use the same review form and response fields but differ in the decision rule translating scientific evidence into an overall rating.Soundness, presentation, and contribution use a 1–4 scale, while the overall rating must be 1, 3, 5, 6, 8, or 10.Source: §G.4 · pp. 59–60
• Reviews must be returned as a single valid JSON object containing numeric scores, written summaries, strengths, weaknesses, questions, ethics status, rating, and confidence.The prompt also defines 1–4 scales for soundness, presentation, and contribution and a 1–5 confidence scale.Source: §G.4 · p. 61
• Strict review judges submissions against accepted top-tier ICLR papers, requiring sufficient novelty, evidence, evaluation, reproducibility, and significance.Technically plausible or well-written papers may still receive reject-level ratings when these standards are unmet.Source: §G.4 · p. 61
• The prompt assigns ratings from 1, 3, 5, 6, 8, or 10 using concrete evidence, starting from 3 = Reject rather than 6.It states that polished writing, confident framing, or strong claims should not earn credit without supporting technical content and evaluation.Source: §G.4 · p. 62
• Upper-bound rules cap ratings when claims lack support, baselines, ablations, evaluation, novelty, or contribution strength, and favor the lower rating when uncertainty remains.Soundness or contribution scores of 1 or 2 generally constrain the overall rating to 1–3 or at most 5, respectively.Source: §G.4 · p. 63