PaperlayerView exact revision on arXiv

Source-linked AI summary

Writing for the Reviewer: Defensive Writing in GPT Models

Junchi Liao

arXiv:2610.11355v1cs.AI

TL;DR

When GPT revises papers, it may narrow or retract authors’ claims, raising whether this corrects overclaiming or anticipates reviewer scrutiny. Models wrote four paragraph types from evidence sheets in separate calls; defensive writing was measured as ungrounded qualifications and claim retreat using sentence coding and claim tracking. Recent GPT versions write defensively, with the behavior tied to anticipated review rather than correction of overclaiming; fewer than one in ten GPT-6-astra retractions are overstated.

  • Problem

    When GPT revises papers, it may narrow or retract authors’ claims, raising whether this corrects overclaiming or anticipates reviewer scrutiny.

    Source: §1 · p. 1

  • Method

    Models wrote four paragraph types from evidence sheets in separate calls; defensive writing was measured as ungrounded qualifications and claim retreat using sentence coding and claim tracking.

    Source: §2.2 · p. 3, §2.4 · p. 3

  • Results

    Recent GPT versions write defensively, with the behavior tied to anticipated review rather than correction of overclaiming; fewer than one in ten GPT-6-astra retractions are overstated.

    Source: §1 · p. 2, §3.3 · p. 7, §4 · p. 8

  • Takeaways & Limitations

    AI reviewers score defensive rewrites higher, whereas human readers find them harder to read, suggesting AI writing and review may amplify the style.

    Source: §1 · p. 2

    The sample covers only four paragraph types in English computer-science papers, so other fields, languages, and full papers remain untested.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Researchers increasingly use ChatGPT to revise their papers, and recent GPT versions often narrow or even retract the authors' claims. We call such changes defensive writing when the given material does not support them, and we test two explanations: the model corrects the authors' overclaiming, or it writes for an anticipated reviewer. We ask GPT versions and models from other developers to rewrite paragraphs from papers written before ChatGPT, or to write from an evidence sheet that lists a paper's method and results. Defensive writing grows with GPT version. GPT-6-astra retracts the authors' claims outright, and when it writes from the evidence sheet, it still adds the most ungrounded qualifications. The results favor the anticipated-review explanation, and correcting overclaiming explains only a small part. When the models are only asked to polish, defense stays near the level of the originals; mentioning review raises it, and one round of self-review raises it further. At the same time, fewer than one in ten of the claims GPT-6-astra retracts are overstated. AI reviewers score defensive rewrites higher, while human readers find them harder to read and the authors less certain. Combining AI writing with AI review may amplify this style.

1 Introduction

The paper defines defensive writing as unsupported narrowing or questioning of authors’ claims and tests whether GPT does this to correct overclaiming or anticipate reviewers. Its results favor the review explanation: recent GPT versions add unsupported qualifications and retract claims, while readers find defensive rewrites harder to read.

Definition and motivation

  • Defensive writing narrows or questions authors’ claims without support in the material provided, unlike caution grounded in evidence.The paper says such qualifications make paragraphs harder to follow and weaken how firmly authors state conclusions.Source: §1 · p. 1

Research question and approach

  • The study tests whether GPT corrects author overclaiming or writes for an anticipated reviewer, using pre-ChatGPT paragraphs and evidence sheets that omit authors’ interpretations.The evidence sheets list a paper’s method, setup, and results; the study tracks unsupported qualifications and whether claims are kept, softened, or retracted.Source: §1 · p. 1

Results

  • GPT-6-astra retracts authors’ claims outright and adds the most ungrounded qualifications from an evidence sheet; fewer than one in ten retracted claims are overstated.AI reviewers score defensive rewrites higher, while human readers find them harder to read and authors less certain.Source: §1 · pp. 1–2

Contribution

  • The authors argue that AI reviewers’ preference for defensive rewrites, contrasted with human readers’ difficulty, may let AI writing and review reinforce the style.The paper defines the measure to distinguish grounded caution from unsupported defense.Source: §1 · pp. 1–2

2 Study Design

The study compares prompted rewrites of pre-ChatGPT paper paragraphs with writing from evidence sheets, using 12 models and measures of unsupported qualifications and claim retreat.

2.1 Rewriting

  • The rewriting sample contains 77 paragraphs from 20 pre-ChatGPT computer-science papers, grouped into novelty, contribution, result-analysis, and discussion types.Models see a paragraph and its location, and are asked to revise it to hold up under peer review; the original serves as the human reference.Source: §2.1 · p. 2

2.2 Writing from Evidence

  • For 16 method papers, evidence sheets retain factual methods, setup, prior work, and exact reported results while omitting interpretations and evaluative language.Models write each paragraph type from a sheet in a separate call; the prompt names a top-tier venue but does not mention review.Source: §2.2 · pp. 2–3

2.3 Writers

  • Twelve writers perform both tasks: eight GPT versions and four control models from other developers.The GPT series ranges from GPT-4o through GPT-6-astra; controls are Claude Sonnet 5, Claude Opus 5.5, DeepSeek-V4-Pro, and Grok 4.7.Source: §2.3 · p. 3

2.4 Measuring Defensive Writing

  • Defensive writing is measured as unsupported qualifications and retreat from authors’ claims, with sentence coding and claim tracking, respectively.Sentence coding checks each sentence against the source material; unsupported labels include not-shown statements, ungrounded doubts, and generic scope restrictions, while grounded limitations are distinguished.Source: §2.4 · p. 3
  • Claim tracking classifies each original claim as kept, strengthened, weakened, scoped, retracted, or dropped, then measures retraction and softening rates.A claim counts as retracted when the rewrite says it does not hold, has not been shown, or applies only more narrowly.Source: §2.4 · p. 3

3 Experimental Results

Defensive writing rises with newer GPT versions and review-oriented prompts, yet most retracted claims are not overstatements. AI reviewers score defensive rewrites higher, while human readers find them harder to read.

3.1 RQ1: Does GPT Write Defensively?

  • From GPT-5.2 onward, the share of authors’ claims softened or scoped exceeds earlier versions, and GPT-6-astra is highest.The share strengthened falls at the same time, and the four controls remain below GPT versions from GPT-5.2 onward.Source: Figure 2 · p. 4, §3.1 · p. 4
  • GPT-6-astra retracts claims in every paragraph type, including at least one in about 40% of result-analysis paragraphs; other models almost never retract claims.Softening rises gradually across the GPT series, while outright negation appears as a jump with GPT-6-astra.Source: §3.1 · pp. 4–5
  • Even with full evidence, GPT-6-astra adds the most ungrounded qualifications alongside the most grounded limitations, with at least one ungrounded qualification in every discussion paragraph.Defense concentrates in result-analysis and discussion paragraphs, followed by novelty, and is weakest in contribution paragraphs.Source: §3.1 · p. 5

3.2 RQ2: What Triggers Defense?

  • When prompts mention review, ungrounded qualifications rise and softened claims increase from about 10% to 15–33%; P5, asking models to address likely reviewer concerns, triggers the strongest defense.Asking to maximize acceptance triggers much less defense, and newer GPT versions respond more strongly to review prompts.Source: §3.2 · pp. 5–6
  • After one review–revise round, three GPT versions add far more ungrounded qualifications and retract many claims, while polish-only chains stay near the originals.Across later rounds, defense stays high rather than receding.Source: Table 2 · p. 5, §3.2 · p. 6

3.3 RQ3: Is the Caution Warranted?

  • Fewer than one in ten claims GPT-6-astra retracts are overstated, while about a third are supported; the authors themselves rarely overstate, and annotators do not confirm the judge’s overstatement labels.The judge labels about 5% of claims overstated or contradicted, and the first prediction that GPT retreats more from overstated claims is weakly supported.Source: §3.3 · p. 7

3.4 RQ4: Reviewers and Readers

  • AI reviewers score P5 rewrites higher than P1 rewrites across all four writers and both reviewer models, and the GPT rewrites draw fewer overclaim criticisms.GPT-6-astra’s P5 rewrites are criticized more often for overcaution but still score above its P1 rewrites; Claude Opus 5.5 is not criticized less for overclaiming.Source: Table 4 · p. 7, §3.4 · p. 8
  • Human readers rate P5 rewrites worse on every reading-experience aspect, especially author certainty, but answer every comprehension question correctly for both versions.Readers also find the rewrites harder to read, less coherent, and more effortful.Source: §3.4 · p. 8, Table 5 · p. 8

4 Conclusion

Recent GPT versions narrow or retract authors’ claims without evidence support, and this behavior is tied to anticipated review rather than correction of overclaiming. AI reviewers score defensive rewrites higher, while human readers find them harder to read.

4 Conclusion

  • Recent GPT versions narrow or retract authors’ claims without support in the evidence, and the behavior is tied to anticipated review rather than correcting overclaiming.Source: §4 · p. 8
  • AI reviewers score defensive rewrites higher, while human readers find them harder to read.Source: §4 · p. 8
  • The findings concern revisions that change the authors’ communicated claims, not merely their wording.Source: §4 · p. 8

Limitations

The paper limits its empirical scope and measurement claims, while situating its approach among work on scientific claim strength and language-model writing and revision. Its source-based comparisons track whether revisions alter communicated commitments without evidence support.

Limitations

  • The sample covers four paragraph types in English computer-science papers, and LLM-based claim coding and two-model review constrain how broadly results should be interpreted.The authors caution that absolute retraction rates are uncertain, model behavior may change with updates, and the reader study included only four readers and GPT-6-astra rewrites.Source: Limitations · p. 9

A.1 Hedging, Stance, and Scientific Claim Strength

  • Prior work distinguishes uncertainty and negation from assertion strength and evidential support; this study brings these questions together by comparing revisions with source material.Source: §A.1 · p. 11

A.2 LLMs for Scientific Writing and Revision

  • Prior systems study collaborative writing and scientific generation, while this paper tests whether reviewer-oriented revision preserves claim strength by tracking commitments against the source across revisions.Source: §A.2 · pp. 11–12

A.3 Automated Peer Review and Evaluation Biases

Automated-review research studies both generated feedback and assigned judgments, while evaluation work raises questions about what model scores measure. The paper separates reviewer criticism of overclaiming from excessive caution and treats scores as judgments under specified instructions, not proof of better-supported claims.

A.3 Automated Peer Review and Evaluation Biases

  • Automated-review systems generate comments and assign judgments, while evaluation studies examine model judgments and their limitations.Source: §A.3 · p. 12
  • Evaluator biases such as position bias, self-preference, verbosity, and self-enhancement are distinct from rewarding defensive qualifications.Source: §A.3 · p. 12
  • The probes compare same-source rewrites and separate criticism of overclaiming from excessive caution; their scores reflect specified review instructions, not better-supported claims.Source: §A.3 · p. 12

B Data and Setup

The study samples 77 paragraphs from 20 pre-ChatGPT papers, drawing from arXiv and EMNLP 2022. Candidate eligibility and source processing shape which paragraphs enter the experiments.

B.1 Papers and Paragraphs

  • Six papers came from stratified arXiv samples in cs.AI, cs.LG, and cs.CV from 2019–2022, and 14 came from EMNLP 2022.The arXiv papers were verified as peer-reviewed; all 20 first arXiv versions appeared between March 2019 and November 2022.Source: §B.1 · p. 12
  • The authors randomly selected 20 candidate papers and sampled one 60–300-word paragraph per available paper and type, yielding 77 paragraphs averaging 128 words.The sampled paragraphs ranged from 72 to 269 words.Source: §B.1 · p. 12
  • Papers had to provide author paragraphs for every evidence-writing task and at least three rewrite paragraph types; citations, cross-references, and equations were replaced with placeholders.Source: §B.1 · p. 12

B.2 Evidence Sheets

The appendix describes how the study builds evidence sheets, configures writers and prompts, and codes rewritten sentences. It also notes a limitation in distinguishing some kinds of unsupported statements.

B.2 Evidence Sheets

  • Kimi-K2.6 builds evidence sheets from related-work, background, method, and experiment sections and tables, returning seven fields for writers.The sheets cover 16 method papers; all 20 source papers have a sheet.Source: §B.2 · p. 12, Table 6 · p. 13
  • The extraction prompt asks for factual records, exact quantitative results, and concrete technical differences, while excluding unsupported interpretations and evaluative claims.It also instructs the extractor to include settings where baselines win, tie, or show small or nonsignificant differences.Source: §B.2 · p. 13

B.3 Writer Settings

  • GPT-5.1 and later GPT versions use high reasoning effort; all GPT models use temperature 1.0 and generate one output per input.Each model has an 8,192-token output limit and otherwise uses provider defaults.Source: §B.3 · p. 13

B.4 Prompts

  • The rewrite prompts prohibit adding results, numbers, or citations absent from the paragraph, while the polish-only prompt asks only for polishing.The polish-only prompt is used for the 12-model comparison and polish-only chains.Source: §B.4 · pp. 13–14
  • Evidence-writing prompts provide a rendered sheet and request text such as an abstract, contribution statement, result analysis, or discussion.Abstracts were generated but are not analyzed in the paper.Source: §B.4 · p. 14
  • Review–revise chains ask a model to review a paragraph and then revise it using the attached review comments.Other prompts ask the writer to withstand peer review or address a reviewer’s comments, while prohibiting new results or citations.Source: §B.4 · pp. 13–14

C.1 Sentence Coder

  • DeepSeek-V4-Flash codes numbered sentences using the writer’s source material, with labels for unsupported doubts, grounded limits, generic scope, claims, and other sentence roles.The labels distinguish statements about what results do not show from factual negative results and supported limitations.Source: §C.1 · pp. 14–15
  • The coder’s not-shown label can include faithful paraphrases of authors’ own statements, and its quote-based split does not reliably distinguish retractions from denials.The main text therefore merges those cases under one label.Source: §C.1 · p. 15

C.2 Coder Validation

The insertion test checks whether the sentence coder recognizes planted sentence types and whether those insertions alter labels on surrounding text. A second coder provides an additional agreement check.

Coder Validation

  • 97% (300/308) of planted retractions, denials, ungrounded doubts, and generic scopes were labeled U, versus 6% (8/133) of planted claims and grounded limitations.The coder labeled 96% (148/154) of planted retractions and denials as ①, but correctly split only 31% of them between the two types.Source: §C.2 · p. 15
  • 93% (2,425/2,604) of the authors’ sentences kept their original labels after a planted sentence was inserted.The insertion test used 441 sentences drawn from six planted types across paragraphs, omitting grounded limitations where none existed.Source: §C.2 · p. 15
  • 97.2% of sentence labels were U under both coders (κ = 0.66), and their writer-level RU values correlated at 0.80; GPT-6-astra had the highest RU under both.On prompt-context rewrites, coder agreement was κ = 0.63.Source: §C.2 · pp. 15–16

C.3 Claim Tracking

Claim tracking records whether original claims survive rewrites and how their strength changes; a separate judge checks claims against evidence sheets. The validation reports both coder agreement and limits on judging overstatement.

C.3 Claim Tracking

  • Claim tracking first lists each paragraph’s claims, then assigns each claim its strongest applicable status in the rewrite, from retracted through kept.The coder is not told which model or prompt produced the rewrite.Source: §C.3 · p. 16
  • GPT-6-astra had the most retracted claims under all three claim-tracking coders and the most softened claims, though only by a small margin under Kimi-K2.6.Disagreements often concerned whether a claim was weakened or retracted.Source: §C.3 · p. 16

C.4 Claim-Support Judge

  • The claim-support judge evaluates each claim against the evidence sheet, counting results and described facts as evidence but not repeated author judgments.The paragraph provides claim context but is not itself evidence.Source: §C.4 · pp. 16–17
  • In a 40-claim human check, students agreed with each other on 30 claims (κ = 0.66) and with the judge on 19 and 21.Both students agreed with the judge on 9 of 12 supported claims but none of 12 claims labeled overstated or contradicted; the oversampled labels do not estimate agreement across all claims.Source: §C.4 · pp. 16–17

D Additional Results

Additional results compare prompt contexts, examine multi-round revision, and report how original claims were classified against evidence sheets.

D.1 Prompt Contexts

  • Table 9 compares polish-only and default prompts, while Table 10 reports the share of claims softened or retracted across seven prompt contexts.The supplied passages describe the tables’ scope but do not provide their cell values.Source: §D.1 · p. 17, Table 9 · p. 17, Table 10 · p. 18

D.2 Multi-round Revision

  • About a third of 12 claims dropped in rounds 2 and 3 were defensive, while about half disappeared during rewrites into definitions and scope statements.The authors exclude dropped claims from their defense measure; about half of the claims dropped in later rounds had already been dropped previously.Source: §D.2 · p. 17

D.3 Claim Support

  • Of 482 original claims, 52% were labeled supported, 5% overstated or contradicted, 15% interpretation, and 28% not covered.Table 12 reports 95% bootstrap intervals for the Table 3 results, resampling paragraphs.Source: §D.3 · p. 18

E.1 AI Reviewer Probes

AI reviewers scored P5 rewrites above P1, while their critiques showed less overclaiming alongside more caution in some GPT-6-astra rewrites. The human-reader study assessed readability, effort, claim identification, author certainty, and comprehension across rewrites.

E.1 AI Reviewer Probes

  • The reviewer comparison used 73 paragraphs with available rewrites, retaining 68 to 72 paragraphs per writer when both reviewers had valid P1 and P5 reviews.Claude Sonnet 5 and DeepSeek-V4-Flash reviewed each version independently using default settings.Source: §E.1 · p. 18
  • Under both reviewer models, P5 rewrites from all four writers scored higher than P1 rewrites on all four aspects.Table 13 reports the mean reviewer scores for P1 and P5.Source: §E.1 · p. 18, Table 13 · p. 19
  • P5 rewrites received slightly fewer weaknesses, mostly for overclaiming, but GPT-6-astra P5 was more often criticized for overcaution and weak significance.Reviewers also found GPT-6-astra P5 less unclear and gave it higher clarity scores, while human readers found those rewrites harder to read.Source: §E.1 · p. 18, Table 14 · p. 19
  • 6.50 to 6.79 against 5.72 to 5.81: among rewrites criticized as overly cautious, GPT-version P5 rewrites still had higher average overall scores than P1.The comparison covers the three GPT versions.Source: §E.1 · p. 18

E.2 Human Reader Study

  • Four readers evaluated 20 result-analysis and discussion paragraphs in P1 and P5 versions, producing 80 readings; each reader saw ten of each version and never both versions of one paragraph.The texts were anonymous, and readers were not told the versions, models, prompts, hypotheses, or earlier results.Source: §E.2 · p. 19
  • Readers rated readability, argument flow, effort, claim identification, and author certainty, then restated the claim and answered comprehension questions.The restatements were not scored; claim identification was readers’ own rating, and comprehension was at ceiling because every answer was correct.Source: §E.2 · pp. 19–20
  • The bootstrap intervals describe variation across 20 paragraphs for these four readers, not variation across readers.The intervals use 2,000 bootstrap resamples of paragraph-level differences.Source: §E.2 · p. 20

F Examples

Figure 5 illustrates how GPT-6-astra’s default-prompt rewrites qualify, retract, or replace claims across four paragraph types. The examples include both generic scope limitations and statements that findings do not establish the authors’ conclusions.

F Examples

  • In the summary example, GPT-5.5 adds only a scope phrase, while GPT-6-astra adds generic-scope and not-shown statements and retracts an explanation.Figure 5 marks generic scope, not-shown statements, and claims labeled retracted by the claim tracker.Source: F Examples · p. 20, Figure 5 · p. 21
  • In the novelty example, GPT-6-astra drops the 30–50% figure and adds four not-shown statements, including one reversing the authors’ conclusion about heuristic datasets.Source: F Examples · p. 20
  • In the contribution example, GPT-6-astra drops the state-of-the-art claim, scopes the baseline comparison to evaluated settings, and questions what the method’s threshold guarantees.Source: F Examples · p. 20
  • In the discussion example, GPT-6-astra replaces the claim that ACML is a general model with a statement that the results do not establish general applicability.Source: F Examples · p. 20
Loading 2610.11355v1…