PaperlayerView exact revision on arXiv

Source-linked AI summary

Style over Substance? Linguistic Bias in LLM-based Peer Review Copy title

Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz

arXiv:2610.35254v1cs.CL

TL;DR

LLMs are increasingly used to support peer review, but it remains important to know whether their assessments reflect research quality or linguistic style. This paper tests content-preserving rewrites across language-proficiency and confidence dimensions. Reviewers consistently favor confident wording, while language-proficiency effects are smaller and more model-dependent.

  • Problem

    LLMs may let linguistic characteristics unrelated to scientific content influence peer-review assessments.

    Source: §1 · pp. 1–2

  • Method

    The authors rewrite papers along language-proficiency and confidence dimensions while preserving content, then evaluate the rewrites with multiple LLM reviewers.

    Source: §1 · pp. 2–3

  • Results

    High-confidence rewrites are preferred 1.3× to 7.0× as often as low-confidence rewrites across reviewers, while language-proficiency preferences are weaker and less consistent.

    Source: §3.1 · p. 6

  • Takeaways & Limitations

    The findings support stress-testing LLM reviewers with content-preserving linguistic rewrites and checking both scores and review arguments.

    Source: §5 · p. 10

    The authors note that identifying influential linguistic features could be misused to manipulate papers for more favorable automated reviews.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload of researchers. At the same time, large language models (LLMs) are increasingly used to support peer review, both by individual reviewers and by conference organizations. While LLM capabilities have improved considerably, their evaluations may be influenced by linguistic characteristics that are unrelated to scientific content. In this work, we introduce a controlled experimental framework for studying such effects by systematically perturbing the linguistic style of research papers. Specifically, we rewrite papers to exhibit (i) linguistic features associated with native or non-native English writing, and (ii) high- or low-confidence linguistic features. To construct these interventions, we derive fine-grained rewriting instructions grounded in the linguistic literature and apply them using LLMs. Crucially, our interventions are designed to preserve the underlying scientific content. We then evaluate the resulting papers using different LLM reviewers. Our experiments expose a critical vulnerability in LLM-based peer review: reviewers systematically reward confident rhetoric independent of scientific content, raising serious concerns about the robustness of their assessments.

1 INTRODUCTION

As LLMs increasingly support peer review, it is important to determine whether they judge research quality or respond to linguistic style. This work introduces content-preserving interventions to test that question and finds that confidence cues strongly affect reviews.

1 INTRODUCTION

  • LLM-assisted peer review is growing, while its judgments may be influenced by properties unrelated to research quality.The paper frames this risk against the substantial time and effort peer review requires.Source: §1 · p. 1
  • The study tests two linguistic dimensions associated with epistemic authority: language proficiency and expressed confidence.The authors argue these features should not substantially affect assessment of underlying research quality.Source: §1 · p. 2
  • The framework rewrites papers along targeted linguistic dimensions while preserving scientific content, then evaluates them with LLM reviewers.Its prompts are grounded in linguistic literature, and the resulting rewrites are checked for content integrity and intended style.Source: §1 · p. 2
  • Confidence has a strong, consistent effect across reviewers, whereas native- versus non-native-associated writing effects are smaller and more model-dependent.Review arguments also differ across rewrites in the aspects discussed and the sentiments expressed.Source: §1 · p. 2
  • The contribution is a controlled framework for stress-testing LLM reviewers using literature-grounded linguistic interventions.Experiments across state-of-the-art reviewers find stronger, more consistent preferences for confidence than for language proficiency.Source: §1 · p. 3

2 METHODOLOGY

The methodology creates content-preserving rewrites along language-proficiency and expressed-confidence dimensions, validates the rewrites, and compares the resulting LLM reviews. It uses a sampled set of ICLR papers and multiple reviewer models, while measuring reviewer score variability.

2 METHODOLOGY

  • The three-stage framework generates paired rewrites, validates content integrity and linguistic style, then compares LLM reviews of the versions.Each reviewer receives one rewritten version at a time and produces a complete review and overall score.Source: §2 · p. 3, §2.3 · p. 5

2.1 LINGUISTIC REWRITING

  • Language-proficiency prompts vary lexical diversity, academic expressions, syntactic variation, and epistemic expressions without portraying lower proficiency as defective English.Both directions aim for plausible publication-quality writing and subtle tendencies rather than stereotypes.Source: §2.1 · p. 3
  • Confidence prompts vary hedging, boosting, modality, and qualification while keeping claim scope, evidence, and scientific meaning unchanged.The rewrites avoid both exaggerated certainty and implausible hesitation.Source: §2.1 · pp. 3–4

2.2 REWRITE VALIDATION

  • Content-integrity measures remain close to their maximum value of 1, and all reported language-proficiency metrics increase for native-associated rewrites.Confidence rewrites contain more boosters and fewer hedges in the intended high-confidence direction.Source: §2.2 · pp. 4–5, Table 2 · p. 6

2.3 COMPARATIVE REVIEWING

  • Reviewers provide strengths, weaknesses, suggestions, four criterion scores on a 1–4 scale, confidence on a 1–5 scale, and an overall rating on a 1–10 scale.The review prompt is based on standard instructions from major AI conferences.Source: §2.3 · p. 5

2.4 CONFIGURATION

  • The evaluation samples 504 anonymized ICLR 2026 papers and uses proprietary and open-weight reviewers selected to avoid known acceptance-decision information.Reviewer variability differs across models: pooled standard deviations are approximately 0.5 points for four models, 0.85 for GLM4.7, and 1.24 for GEMINI 3.5 FLASH.Source: §2.4 · p. 5, Table 3 · p. 6

3 EXPERIMENTAL RESULTS

Across 504 papers, LLM reviewers show a substantially stronger and more consistent preference for high-confidence rewrites than for native-associated writing, including confidence-favoring decision flips. Combined comparisons and aspect-level analyses reinforce the confidence effect, while simulated style-biased reviewers show the framework can detect proficiency preferences when present.

3.1 MAIN RESULTS

  • The experiments examine pairwise score preferences, average ratings, and acceptance decision flips for native versus non-native and high- versus low-confidence rewrites across 504 papers.The combined-style comparisons use 126 papers.Source: §3.1 · p. 6, Table 4 · p. 7, §3.2 · p. 7
  • High-confidence rewrites are preferred 1.3×–7.0× as often as low-confidence versions across all reviewers, while native rewrites receive higher scores more often for 5/6 reviewers at only 1.0×–1.3×.Average scores are nearly identical for native and non-native rewrites, but consistently higher for high-confidence rewrites.Source: §3.1 · p. 6
  • For confidence, decision flips predominantly favor high-confidence versions; CLAUDE SONNET 4.6 accepts the high-confidence rewrite and rejects the low-confidence rewrite in 21.0% of reviews, versus 2.8% in the reverse direction.Language-proficiency flips are largely balanced; flips are rare for GEMMA3 and QWEN3 VL because their average scores are well above the acceptance threshold.Source: §3.1 · p. 6

3.2 IN-DEPTH ANALYSIS

  • When confidence is reversed in combined rewrites, all reviewers favor the non-native but high-confidence version; confidence dominates these comparisons, with preference ratios up to 9.1× for native/high-confidence over non-native/low-confidence.Conditional comparisons largely reproduce the isolated effects, and proficiency differences remain relatively small when confidence is fixed.Source: §3.2 · p. 7
  • With linguistic-style weighting added to reviewer prompts, simulated biased reviewers show statistically significant native-associated writing preference ratios of 6.8 for GPT-5.5, 4.3 for CLAUDE SONNET 4.6, and 2.2 for GEMINI 3.5 FLASH.The analysis uses 126 papers and supports the framework’s ability to reveal both presence and absence of systematic preferences.Source: §3.2 · p. 7
  • The aspect analysis uses 16 coarse-grained review aspects, automated positive/negative annotations, and within-paper differences in net polarity averaged across papers.Figure 2 shows the top 10 aspects with the largest absolute differences and 95% bootstrap confidence intervals.Source: §3.2 · pp. 8–9, Figure 2 · p. 8
  • Native-associated writing receives less criticism and more praise for Presentation (−2.4 pp and +3.0 pp), raising its net polarity by 0.16; Presentation also has the largest confidence-related net-polarity gain (+0.29).These Presentation shifts do not systematically affect review scores (r=0.04).Source: §3.2 · p. 9
  • For high-confidence rewrites, Methodology, Theory, and Contribution are perceived more positively alongside Presentation, and overall ratings correlate with the share of positive aspect mentions (r=0.81).These changes contribute to the increase in positive-mention share and the preference ratios in Tables 4 and 5.Source: §3.2 · p. 9

4 RELATED WORK

Related work documents linguistic differences in academic writing and evidence that linguistic presentation can affect human review, while also identifying reliability and fairness concerns in LLM reviewing. This study complements prior observational and counterfactual work with controlled, linguistically grounded transformations of the same papers.

LLM peer review

  • Prior studies report LLM-reviewing biases tied to author properties and vulnerability to adversarial text, while Kousha & Thelwall (2026) find linguistic complexity and length more associated with ChatGPT research evaluations than expert evaluations across many fields.This work complements those findings by manipulating proficiency and confidence in the same papers while preserving research content.Source: §4 · p. 9

Linguistic biases in human peer review

  • Human-review research reports L1/L2 differences in lexical, phraseological, and epistemic expression, and preliminary evidence that identical content receives lower ratings in non-native-like English.The controlled study was limited to paper abstracts; an observational study found lower priority scores and acceptance rates for submissions from non-English-speaking countries.Source: §4 · p. 9
  • Prior work also associates writing style with publication recommendations in simulated review, motivating a controlled test of whether linguistic presentation affects LLM peer review.The paper focuses on language proficiency and expressed confidence as dimensions of linguistic presentation.Source: §4 · p. 9

5 CONCLUSION

The paper presents a controlled framework for investigating linguistic bias in LLM-based peer review and reports substantial preference for high-confidence rewrites, including decision flips. It recommends stress-testing reviewer stability under content-preserving linguistic changes and cautions that raw scores from individual reviewers may not be directly comparable without calibration.

5 CONCLUSION

  • The framework varies linguistic markers of epistemic authority while preserving scientific content, and experiments find substantial preference for high-confidence rewrites that can produce accept–reject decision flips.Review aspects also vary in frequency and sentiment with linguistic features.Source: §5 · p. 9
  • The authors recommend stress-testing reviewers on content-preserving rewrites across multiple linguistic dimensions, assessing both scores and review arguments, and checking whether confidence is treated as evidence or scientific quality.They also caution that individual reviewers’ raw scores may require calibration before comparison.Source: §5 · p. 10

AI USE STATEMENT

  • Generative AI supported experiment design, coding, review generation, linguistic rewrites, literature analysis, and manuscript editing; the authors reviewed these activities and accept responsibility for the work.The authors state that AI tools were not used to formulate mathematical claims or assist mathematical proofs.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study does not advocate replacing human reviewers; it warns that linguistic variation associated with researchers’ backgrounds may influence LLM-generated review outcomes despite preserved research content.The authors identify potential dual use: exploiting influential linguistic features to seek more favorable automated reviews, and motivate careful evaluation before high-stakes use.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper describes its processing pipeline, models, settings, sampled papers, and validation measures, and states that code, paper lists, rewrites, and generated reviews will be released upon publication.Reviewer variance across repeated generations is quantified because several models do not support fixed sampling parameters.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The implementation converts papers into structured text, blinds them, and rewrites them section by section without context. Confidence is measured using booster and hedge frequencies, and the same default sampling settings are used across conditions.

Paper processing

  • PDFs are converted to structured text with MinerU, cut after references, and blinded by removing titles, author and affiliation information, venue watermarks, page numbers, and line-number gutters.Figures are not sent to models, although captions remain as text.Source: A IMPLEMENTATION DETAILS · p. 14
  • Rewriting is performed section by section without contextual input, to preserve paper organization and avoid model context or output limits while limiting rewrites to stylistic changes.Source: A IMPLEMENTATION DETAILS · p. 14

Confidence Indicators

  • Confidence is quantified as booster and hedge frequencies per 1,000 words using Hyland’s metadiscourse taxonomy, with categories mutually exclusive to avoid double-counting.Papers are segmented into chunks and terms are counted using whole-word matching.Source: A IMPLEMENTATION DETAILS · p. 14

Generation settings

  • All rewrites use Claude Opus 5, and rewriting and reviewing use each model’s default sampling configuration without fixed temperature so models remain more comparable.GLM 4.7 has reasoning enabled; other models use their defaults, and reviews are generated in a single call.Source: A IMPLEMENTATION DETAILS · p. 14

B FURTHER REWRITE EXAMPLES

Tables 6 and 7 provide further examples of rewrites from the methodology and results sections, produced using the same process as Table 1.

B FURTHER REWRITE EXAMPLES

C PROMPT SENSITIVITY

The prompt-sensitivity test reran CLAUDE SONNET 4.6 on 126 papers using neutral prompt variations and an explicitly style-biased prompt. Neutral variations left the main conclusions unchanged, while the biased prompt produced a native-writing preference.

C PROMPT SENSITIVITY

  • CLAUDE SONNET 4.6 was rerun on Batch 1 (126 papers) with Brief, Minimal, and Bare neutral prompt variations and an explicitly style-biased prompt.The neutral versions progressively removed reviewer responsibilities, criteria, rating-scale anchors, fair-feedback guidance, and per-criterion subscores.Source: C PROMPT SENSITIVITY · p. 15
  • The native-versus-non-native null result became a 4.3× preference for native writing under the explicitly biased prompt (p < 0.0001).Under every neutral prompt, the native-versus-non-native contrast remained null and the high-versus-low-confidence contrast remained large and significant (p < 0.0001).Source: C PROMPT SENSITIVITY · p. 15

D REWRITE EFFECTS ON REVIEW CRITERIA

Language-proficiency and confidence rewrites affect review criteria differently: confidence shifts several assessments, while proficiency differences are concentrated in Clarity. Review scores also align with extracted review aspects, and paired comparisons distinguish systematic rewrite effects from generation variability.

D REWRITE EFFECTS ON REVIEW CRITERIA

  • Confidence raises Soundness, Significance, and Evidence for all six reviewers; native-associated writing raises Clarity ratings for four reviewers by 1.3–2.1×, without changing overall ratings.High-confidence writing is also rated clearer by CLAUDE SONNET 4.6 (10×), GLM4.7 (5×), and GPT-5.5 (2.1×), while three other reviewers show no significant Clarity effect.Source: D REWRITE EFFECTS ON REVIEW CRITERIA · pp. 15–16, Figure 3 · p. 17
  • Negative Presentation mentions correlate with Clarity (r=−0.42), while positive Result mentions correlate with Evidence.Soundness correlates with positive Theory (r=+0.25) and negative Methodology mentions (r=−0.22); Significance correlates modestly with positive Novelty (r=+0.19) and Contribution (r=+0.21).Source: D REWRITE EFFECTS ON REVIEW CRITERIA · p. 16
  • For overall ratings, 88% of native/non-native pairs and 85% of confidence pairs differ by at most one point; fewer than 4% of criterion-score pairs differ by two or more.Differences of two or more points cluster in GEMINI 3.5 FLASH and GLM4.7, which also have the highest repeated-generation variance.Source: D REWRITE EFFECTS ON REVIEW CRITERIA · p. 16
  • GEMINI 3.5 FLASH and GLM4.7 show nearly identical mean absolute score differences across rewrite dimensions, yet only confidence rewrites induce a systematic preference.Paired rewrites of the same paper reveal stylistic biases even when individual reviewer scores fluctuate considerably.Source: D REWRITE EFFECTS ON REVIEW CRITERIA · p. 17

E POLARITY OF REVIEW ASPECTS PER LLM REVIEWER

The section’s materials describe per-model paired shifts in positive and negative review-aspect mentions across the two linguistic dimensions, alongside prompts for review generation and aspect annotation.

E POLARITY OF REVIEW ASPECTS PER LLM REVIEWER

  • Figures 4 and 5 present per-model paired prevalence shifts in aspect mentions for native versus non-native and high- versus low-confidence rewrites.Figure 5 specifies shifts in percentage points for positive and negative mentions.Source: E POLARITY OF REVIEW ASPECTS PER LLM… · p. 17, Figure 5 · p. 18
  • The review-generation prompt specifies review criteria, a Likert scale, reviewer responsibilities, and an output format for integration into the framework.A separate prompt simulates a reviewer biased toward linguistic style and presentation.Source: Figure 6 · p. 19, Figure 7 · p. 20
  • The aspect-annotation prompt asks annotators to extract every distinct evaluative mention of a peer-review aspect from review prose.Figures 8–16 provide rewriting prompts for native-associated, non-native-associated, high-confidence, low-confidence, and their interactions.Source: Figure 8 · p. 20, Figure 9 · p. 21, Figure 10 · p. 22
Loading 2610.35254v1…