PaperlayerView exact revision on arXiv

Source-linked AI summary

How Robust Is Multimodal Claim Verification to LLM Rewriting?

Yun-Ang Wu, Xanh Ho, Andre Greiner-Petter, Sunisth Kumar, Tian Cheng Xia, Florian Boudin, Akiko Aizawa

arXiv:2610.02841v1cs.CL

TL;DR

The paper asks how stylistic changes affect scientific claim verification, a question complicated by prior review-score studies that combine wording with other factors. It compares verdicts for original and rewritten claims using natural rewriting and controlled vocabulary injection. Accuracy remains robust, although probability shifts are consistent under some conditions, especially hedging-oriented ones.

  • Problem

    Prior work reports that rewriting can inflate paper-review scores, but review scores conflate wording with argument structure, novelty, and presentation, leaving its effect on single-claim verification unclear.

    Source: §1 · p. 1

  • Method

    The study compares verification of original and rewritten claims against the same evidence, using natural paraphrasing and controlled injection of single vocabulary items.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2.2 · p. 5

  • Results

    Accuracy remains robust across model sizes and families, while cautious-tone rewriting and conservative-word injection each produce significant ∆pref shifts in 11 of 11 models.

    Source: §6 · p. 8, §5 · p. 8

  • Takeaways & Limitations

    Fact-checking appears harder to sway through claim style alone than paper-review prediction, although some rewriting conditions shift model probabilities.

    Source: §6 · p. 8

    The main experiments use one rewriting model and one dataset, limiting generalizability to other rewriters, domains, and evidence types.

    Source: §7 · p. 9

Abstract

from arXiv · show

LLMs are known to introduce stylistic changes into generated text, yet how these stylistic shifts affect model decisions on scientific tasks remains underexplored. In this paper, we focus on multimodal claim verification, where the goal is to determine whether a textual claim is grounded in a given piece of evidence. We apply two rewriting strategies: natural rewriting, which simulates how researchers routinely use LLMs to polish academic text, and controlled injection, which inserts a single LLM-associated word to isolate the effect of vocabulary choice. We evaluate 11 open-weight models spanning five VLM families and ranging from 2B to 38B parameters. We find that models are robust to these modifications: most show no significant drop in accuracy, and compared to prior work on review-score manipulation, verification appears far more stable. However, consistent probability shifts do occur. Hedging-oriented conditions produce significant shifts across nearly all models, while boosting conditions show a weaker effect and general polishing conditions (e.g., grammar correction, fluency improvement) have little effect.

1 Introduction

LLMs increasingly shape scientific writing and may also evaluate text they helped produce. This paper isolates wording changes in multimodal claim verification, finding stable accuracy but consistent probability shifts under some rewriting conditions.

1 Introduction

  • LLMs are increasingly used to shape scientific writing, while conferences are exploring LLM-assisted review pipelines.The passage notes that this could put LLMs in the position of evaluating text that LLMs helped produce.Source: §1 · p. 1
  • The study tests whether stylistic changes that preserve claim meaning shift verification behavior, distinct from semantically meaningful intensity changes.Figure 1 frames the question around changes in p(support) when claim intensity changes.Source: Figure 1 · p. 1
  • Paper-level review scores can be influenced by rewriting, but confound wording with argument structure, novelty, and presentation.The paper instead focuses on single-claim verification against visual evidence, where wording is the variable under study.Source: §1 · p. 1
  • Models across sizes and families remain robust in accuracy, yet certain rewriting conditions produce consistent probability shifts.The study evaluates 11 models across five families.Source: §1 · p. 2
  • The evaluation uses natural LLM paraphrasing and controlled single-word injection to study stylistic variation in multimodal claim verification.Natural rewriting simulates routine academic polishing, while injection isolates individual vocabulary choices.Source: §1 · p. 2

2 Related Work

Related work documents LLM use in scientific writing and review-score manipulation, alongside research on multimodal verification and sensitivity to surface-level changes. This paper focuses on naturalistic rewriting in claim verification.

2.1 LLM-Assisted Scientific Writing

  • Studies report increased LLM-assisted scientific writing and characteristic vocabulary, including growth in usage and hyperbolic style trends.Reported examples include LLM assistance in scientific abstracts and an increase in style-word usage after the LLM era.Source: §2.1 · p. 2

2.2 LLM-Assisted Adversarial Rewriting

  • Prior work finds that adversarial rewriting or paraphrasing can improve LLM review scores without changing paper semantics.Reported approaches include character-, word-, and sentence-level attacks and LLM-based paraphrasing.Source: §2.2 · p. 2

2.3 Multimodal Claim Verification

  • Multimodal scientific verification grounds claims in paper figures or tables; SciClaimEval creates refuted pairs by modifying evidence rather than claims.Table 1 illustrates natural paraphrasing alongside controlled insertion of a single LLM-associated word.Source: §2.3 · pp. 2–3, Table 1 · p. 3

2.4 Sensitivity to Surface-Level Perturbations

  • Prior studies report label changes from meaning-preserving rewrites, prompt-format sensitivity, and effects of superficial stylistic changes; this work studies naturalistic rewriting in verification.The cited examples include sentiment classification and semantic-agnostic biases.Source: §2.4 · p. 3

3 Methodology

The methodology compares verdicts for original and rewritten claims against the same visual evidence. It uses realistic academic-editing prompts and controlled word insertion to examine how rewriting affects verification.

3.1 Multimodal Claim Verification

  • For a claim and scientific figure or table, the task predicts support or refute; rewriting changes the claim while evidence stays fixed for verdict comparison.The method compares V(c, e) with V(c′, e).Source: §3.1 · p. 3

3.2 Rewriting Strategies

  • The study pairs natural rewriting with controlled rewriting, simulating realistic use cases in one strategy and inserting a single word in the other.Table 1 gives rewrite examples, while Table 2 reports accuracy and F1 against original, unmodified claims.Source: §3.2 · pp. 3–4, Table 2 · p. 4

3.2.1 Natural Rewriting

  • Natural rewriting is organized into five categories reflecting common motivations for polishing academic text: language polishing, fluency improvement, academic style alignment, outcome-aware reviewer polishing, and cautious tone.Source: §3.2.1 · p. 4
  • Human review found 81.8% of sampled rewrites semantically consistent, while annotators disagreed on which cases crossed the boundary.The 282 claims covered conditions OA, CT, C1, and D1; agreement was κ = 0.25 and Gwet’s AC1 = 0.71.Source: §3.2.1 · p. 5

3.2.2 Controlled Rewriting

  • Controlled rewriting isolates vocabulary by injecting decorative or conservative words; the pools contain 114 decorative and 39 conservative words.The conservative pool includes 31 added hedging terms; neutral words are excluded because they are not expected to shift the style signal.Source: §3.2.2 · p. 5
  • Each controlled injection adds exactly one selected word while requiring the original claim’s words and factual content to remain unchanged.Candidates are sampled from the selected pool to encourage diverse but grammatically natural insertions.Source: §3.2.2 · pp. 5–6

4 Experimental Setup

The experiments use paired scientific claims and visual evidence to isolate rewriting effects, evaluating 11 open-weight VLMs with accuracy, F1, and probability-shift measures. Statistical tests use paired bootstrapping with joint false-discovery-rate correction.

4.1 Datasets

  • SciClaimEval pairs each claim with supporting and refuting evidence images, so the same claim wording is compared across verdicts and modification effects are less confounded by claim differences.Unpaired instances are discarded after filtering.Source: §4.1 · p. 6

4.2 Models

  • The evaluation covers 11 models from five open-weight VLM families, spanning 2B–38B parameters, with gpt-oss-120b generating rewrites.At inference, 10 reasoning paths are sampled per data point and model; the support-prediction fraction estimates p(support).Source: §4.2 · p. 6

4.3 Metrics.

  • The study reports pair-level F1 and stricter paired accuracy, alongside probability-shift patterns that classify changes as globally optimistic, globally conservative, sharper, or duller.The paired-bootstrap tests use 1,000 resamples and Benjamini–Hochberg FDR correction jointly across 154 model-condition-metric cells per table.Source: §4.3 · p. 7

4.4 License.

  • SciClaimEval is released under CC BY 4.0, and the rewriting and evaluation models are open-weight under their respective licenses.Source: §4.4 · p. 7

5 Experimental Results

Rewriting rarely changes verification accuracy, although hedging conditions produce consistent probability shifts. Most shifts do not change verdicts because predictions are typically far from the decision threshold.

RQ1: Does rewriting change the verdict?

  • 0.610 → 0.557 (q < 0.001) under CT for gemma-4-31B-it; significant accuracy or F1 drops occur in only a few models, including InternVL3-8B under CT.The reported drops are concentrated rather than consistent across models or strategies.Source: §5 · pp. 7–8
  • Unlike prior review-score manipulation, this study uses non-adversarial rewriting and single-claim visual grounding; whether adversarial rewriting can alter verification remains open.Source: §5 · p. 8

RQ2: Does rewriting change confidence?

  • CT and C1 each produce significant ∆pref shifts in 11 of 11 models, while OA produces significant ∆psup shifts in 6 of 11 and general polishing conditions show little probability movement.Boosting’s effect on ∆pref is mixed; hedging shifts persist after filtering certain insertions and with a second rewriter.Source: §5 · p. 8
  • Larger Qwen3-VL models show larger probability drops under OA and increases under CT, but the relationship between model size and shifts is not clear overall.The family-specific pattern is not universal across model families.Source: §5 · p. 8

RQ3: Why do probability shifts not translate to accuracy changes?

  • Most base predictions cluster near 0 or 1, so significant 0.03–0.05 shifts often do not cross the 0.5 decision boundary; flip rates are mostly below 15%.The authors suggest stylistic effects on verdicts may be larger in more difficult or subjective cases closer to 0.5.Source: §5 · p. 8

6 Conclusion

Across model families and sizes, stylistic rewriting leaves accuracy robust while producing probability shifts under some conditions. The paper attributes the difference to decisive baseline predictions and identifies broader generalization as future work.

  • Models across different sizes and families are robust to stylistic rewriting in accuracy metrics, although some rewriting conditions produce consistent probability shifts.Source: §6 · p. 8
  • Probability shifts often leave accuracy unchanged because predictions cluster near decisive probabilities, making the changes ineffective after discretization.Source: §6 · p. 8
  • The paper concludes that claim verification appears harder to sway by changing claim style than paper-review prediction, while generalization beyond reviewing and claim verification remains a question for future work.It also identifies separating semantic from stylistic change as a need for more rigorous analysis.Source: §6 · p. 8

7 Limitations

The authors identify limits in separating stylistic from semantic changes and in generalizing the evaluation and confidence estimates.

7 Limitations

  • Hedging before numerical values can reduce claim precision, and rule-based filtering does not capture every semantic change.The authors note that even human annotators often disagree about whether rewrites preserve meaning.Source: §7 · pp. 8–9
  • The main experiments use one rewriting model and SciClaimEval, limiting generalizability to other rewriters, domains, and evidence types.The second-rewriter check covers only four conditions, one verifier, and the dev set.Source: §7 · p. 9
  • Some SciClaimEval claims may have been polished by LLMs before this study, potentially dampening the observed rewriting effects.Source: §7 · p. 9
  • Support probabilities come from 10 sampled reasoning paths in 0.1 increments; shifts may depend on this unbenchmarked estimation method.The authors leave comparison with token-level log probabilities and verbalized confidence to future work.Source: §7 · p. 9

8 Ethical Considerations

The study uses published scientific claims and evidence, reports its data and model licensing, and describes the annotation and AI-assistance practices.

8 Ethical Considerations

  • The datasets contain claims and evidence from published open-access papers, without personally identifying information or offensive content.Source: §8 · p. 9
  • NLP researcher co-authors annotated semantic consistency without separate compensation; AI-assisted outputs were reviewed, and code and data are publicly available.The authors also acknowledge that the rewriting techniques could in principle be used to manipulate automated verification systems.Source: §8 · p. 9

A Robustness to the Choice of Rewriter

The additional experiments find broadly similar confidence drift with another rewriter and show that the observed pattern persists after filtering potentially meaning-altering rewrites.

A Robustness to the Choice of Rewriter

B Semantic Change of Claims after Rewriting

  • After filtering hedges near numerical or comparative expressions, unflagged ∆psup and ∆pref remain similar to or slightly below full-set shifts.The filter flags 26.3% of CT rewrites and 22.9% of C1 rewrites, and fewer than 10% in other conditions.Source: Table 6 · p. 11, B Semantic Change of Claims after… · p. 11

C Prompts

  • The appendix provides natural- and controlled-rewriting templates, with five prompt variants per category and five named natural-rewriting categories.The categories are Language Polishing, Fluency Improvement, Academic Style Alignment, Outcome-Aware, and Cautious Tone.Source: C Prompts · p. 12, Figure 4 · p. 12

D Flip Rates Across All Models

E Annotation Guidelines

F Word Lists

G Data Exclusions

The appendix describes exclusions used to preserve paired evaluations and reports human judgments of whether rewrites preserve claim meaning.

G Data Exclusions

  • Malformed controlled rewrites and unparseable model responses are excluded, with paired items dropped together and exclusions consistent across conditions.The number excluded is small relative to the 788-pair evaluation set.Source: G Data Exclusions · p. 12

H Human Annotation

  • Four NLP researchers judged whether sampled original and rewritten claims had the same meaning, with changed words highlighted and conditions hidden.Each claim was assigned to two annotators, but incomplete assignments left 391 judgments across 282 claims.Source: H Human Annotation · pp. 12–13
  • Annotators judged meaning preserved in 81.8% of judgments; agreement was 78.9%, with Fleiss’ κ of 0.251 and Gwet’s AC1 of 0.706.Per-condition agreement estimates warrant caution because only 26–29 doubly annotated claims contributed to each condition.Source: H Human Annotation · p. 13, Table 7 · p. 13

I Qualitative Examples

Examples for Qwen3-VL-8B-Instruct show both large confidence shifts that flip predictions and rewrites that leave predictions unchanged. The examples span natural rewriting and controlled conditions, including polishing and hedging.

I Qualitative Examples

  • A controlled rewrite shifted p(support) from 0.0 to 0.9 (∆p = +0.9) and flipped a supported claim’s prediction.The original and rewritten claims both compare bacterial categories in clinical samples and cultures.Source: I Qualitative Examples · p. 14
  • In Case 2 (high shift), under OA, the Gold label was Supported; natural rewriting changed p(support) from 0.9 to 0.0 (∆p = −0.9), flipping the prediction.Source: I Qualitative Examples · p. 14
  • Formality polishing left p(support) unchanged at 0.6, with no prediction flip.The rewrite changes phrasing while retaining the claim about CG and fine-tuning for formality control.Source: I Qualitative Examples · p. 14
  • A hedging insertion also left p(support) unchanged at 0.6, with no prediction flip.The rewrite adds “seemingly” to a claim about AD-Drop’s improvements.Source: I Qualitative Examples · p. 14
  • Table 8 presents representative high-shift and no-shift cases, with p estimated from 10 sampled answers before and after rewriting.The examples are for Qwen3-VL-8B-Instruct and use evidence images from SciClaimEval.Source: I Qualitative Examples · p. 13, Table 8 · p. 14

Related papers

Loading 2610.02841v1…