Source-linked AI summary
Towards Robust Numerical Claim Verification
Peter Røysland Aarnes, Vinay Setty
TL;DR
Numerical claim verification remains brittle, with small value perturbations exposing failures even in frontier models. This paper adversarially fine-tunes Qwen3 models on perturbed claims and reports improved robustness, including gains across model scales and transfer settings.
Problem
Even frontier models can fail when numerical values in claims are slightly perturbed, and benchmark-style numerical reasoning does not establish robust evidence-grounded verification.
Source: §1 · p. 1Method
The paper applies adversarial fine-tuning to Qwen3 models using numerically perturbed claims and evaluates robustness across perturbations, domains, and languages.
Source: §1 · p. 1Results
M0.6B All-Perturbs beats zero-shot M32B Base, GPT-5.4 Pro, and Gemini 2.5 Flash on T→F by at least 14 pp in micro-average accuracy.
Source: §6 · p. 9Takeaways & Limitations
The findings support a generalised numerical decision boundary rather than memorised edits, with robustness transferring across languages and domains without target-language data.
Source: §7 · p. 9Experiments use only the Qwen3 family and assume gold evidence, so robustness across other model families and under noisy or incomplete retrieval remains unconfirmed.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) are widely used for claim verification, yet remain brittle for numerical reasoning: even small changes in value can sharply degrade accuracy. We show that this brittleness persists in frontier LLMs, but can be mitigated through adversarial fine-tuning on numerically perturbed examples. Using parameter-efficient fine-tuning, small Qwen3 models (0.6B$\unicode{x2013}$8B) reach 98.7% accuracy on label-flipping perturbations, outperforming larger zero-shot models and frontier systems (GPT-5.4 Pro (74.0%) and Gemini 2.5 Flash (73.9%)). The gains generalise to unseen perturbation types, indicating robust numerical decision boundaries rather than memorised edits. Robustness also transfers without target-domain data, significantly improving cross-lingual performance in Spanish. We further show that the same fine-tuning recipe confers robustness to evidence-side perturbations, using the VitaminC dataset.
1 Introduction
Numerical claim verification remains brittle even for frontier models, as small changes in claim values can reverse labels and expose failures of surface-form matching. The paper studies adversarial fine-tuning on numerical perturbations as a way to improve this robustness.
1 Introduction
- Small numerical changes can sharply degrade verification because models may match lexical surface forms instead of reconciling claim values with evidence.The introduction notes that numerical claims require interpreting context such as time spans, units, and denominators.Source: §1 · p. 1
- The study applies six numerical perturbation categories to QuanTemp and fine-tunes Qwen3 models from 0.6B to 8B with QLoRA and soft prompt tuning.It evaluates generalisation on English and Spanish corpora unseen during training.Source: §1 · p. 1
- 97.7% versus 74.0% micro-average accuracy on label-flipping perturbations: adversarially fine-tuned Qwen3-8B outperforms GPT-5.4 Pro.The paper also reports that fine-tuned models outperform the zero-shot baseline on Spanish, while the AVeriTeC accuracy difference is not significant and gains there are in coverage.Source: §1 · p. 2
- The paper highlights that Masked and Numeration perturbations are essential, Approximation and Random are redundant, and English-trained robustness transfers to Spanish without target-language training data.It also reports that unperturbed training examples shift the decision boundary and reduce label-preserving accuracy.Source: §1 · p. 2
2 Related work
Prior work documents numerical brittleness across verification and reasoning tasks, while adversarial training and parameter-efficient adaptation offer related approaches. The paper applies these ideas to numerical claim verification.
2 Related work
- Prior numerical-claim verification work includes QuanTemp and a perturbation taxonomy, while FEVER and retrieval-to-program-guided methods are broader fact-checking benchmarks and approaches.The passage cites V et al. (2024) for QuanTemp and Aarnes and Setty (2025) for the taxonomy.Source: §2 · p. 2
- Numerical perturbations can change model predictions across QA, claim verification, and mathematical reasoning, while multilingual surface variation adds an attack surface not addressed by monolingual robustness studies.The cited examples include differences in decimal separators, numeral systems, and written-out forms.Source: §2 · p. 2
- Adversarial training has improved robustness in other settings, including worst-case augmentation, gradient-based NLP perturbations, math adversaries, and counterfactually revised or adversarially collected data.This paper applies the strategy to numerical claim verification across six perturbation types.Source: §2 · p. 2
- Parameter-efficient methods such as LoRA, QLoRA, and soft prompts make model adaptation practical, but the authors report no prior study applying adversarial fine-tuning to numerical perturbations in claim verification.Earlier fact-checking studies also report fine-tuned models outperforming larger zero-shot LLMs.Source: §2 · p. 2
3 Methodology
The methodology generates numerical claim perturbations and trains models on mixtures of original and perturbed examples. It compares parameter-efficient adaptation strategies and evaluates robustness across perturbation configurations and held-out settings.
3 Methodology
- The methodology has two steps: generate adversarial numerical-claim perturbations, then fine-tune models on a mixture of perturbed and original data.The perturbation approach follows related taxonomies and methods.Source: §3 · pp. 2–3
3.1 Generating Numerical Perturbations
- Numerical values in QuanTemp claim–evidence pairs are tagged and transformed using six types, with value-preserving and value-changing variants for Numeration, Approximation, and Range.Random replacement, Negation, and Masking produce only false variants; perturbations are generated within each predefined split to prevent cross-split leakage.Source: §3.1 · p. 3
- Held-out generalisation is evaluated on AVeriTeC 2.0 dev and a Spanish CheckThat! 2025 test set.The passages report 119 numerical claims for AVeriTeC and 2,633 claims for the Spanish test set.Source: §3.1 · p. 3
3.2 Adversarial Fine-tuning
- QLoRA trains low-rank residuals in selected projections of a frozen 4-bit backbone, whereas soft prompt tuning trains prepended vectors and a veracity head with backbone weights fixed.Both strategies are compared as parameter-efficient approaches to the same verification task.Source: Figure 2 · p. 3, §3.2 · pp. 3–4
- Adversarial fine-tuning treats verification as binary classification and optimises binary cross-entropy on the union of original examples and perturbed claims with revised labels.The perturbed claim is paired with evidence, and its label is revised for the perturbation.Source: §3.2 · p. 3
3.2.1 Training Data Sampling
- Four training configurations use original-only, leave-one-out, single-perturbation, or all-perturbation data, each balanced between original and perturbed examples.The validation set retains its natural distribution.Source: §3.2.1 · p. 4
3.3 Evaluation and Fine-tuning
- Evaluation includes base-model prompting comparisons, larger and proprietary zero-shot references, and smaller models fine-tuned with all perturbations.The Numerical Counterexample Prompt is omitted from the main evaluation because of high parse-failure rates.Source: §3.3 · p. 4, §3.4 · p. 4
4 Experimental Setup
The experiments use Qwen3 models and zero-shot closed-source frontier baselines, with parameter-efficient adaptation and a controlled evaluation protocol. Gold evidence isolates verification performance from retrieval quality, and results are reported from single runs.
4.1 Model Set and Inference Settings
- Qwen3-8B is the primary model, with experiments across Qwen3 sizes from 0.6B to 32B and zero-shot GPT-5.4 Pro and Gemini 2.5 Flash baselines.The Qwen3 family enables scale comparisons under a consistent pre-training recipe; the authors caution that quantitative effects may differ across model families.Source: §4.1 · p. 4
4.2 Parameter-Efficient Adaptation
- Models train for two epochs with a 1:1 ratio of original to perturbed examples; soft prompt tuning uses 20 learnable vectors while other hyperparameters match QLoRA.The passage specifies AdamW with an 8-bit optimizer and a learning rate of 1×10^-4.Source: §4.2 · p. 4
4.3 Evaluation
- Accuracy is measured per perturbation type against corresponding original examples, using strict JSON-label scoring; gold evidence isolates verification from retrieval errors.Results are single-run because repeating inference across more than 20 model configurations was prohibitively expensive; the authors report large, consistent QuanTemp effects and significance tests for several comparisons.Source: §4.3 · pp. 4–5, Table 2 · p. 5
5 Results
Results vary by perturbation direction: adversarially fine-tuned models improve performance on claim-side perturbations and Spanish evaluation, but claim-side training does not transfer to evidence-side revisions; joint training performs well on both.
5.1 True →False
- All-Perturb training reaches near-perfect accuracy on Negative and Masked label-flipping perturbations, while single-type training collapses on unseen types.The leave-one-out group has the best accuracies overall, but omitted-type performance collapses for Numeration, Negation, and especially Masked; Masked detection must be trained directly.Source: §5.1 · pp. 5–6
5.2 True →True
- The Base Group has the smallest accuracy fluctuations on label-preserving edits, with MOnly-Original at mid-90s accuracy (±0.57%).All-Perturb models also generally show small fluctuations between original and perturbed examples.Source: §5.2 · p. 6
- On label-preserving perturbations, M0.6B All-Perturbs exceeds M32B Base across all perturbation types, while MExcl-Random has the strongest overall profile.Single-type training provides no immunity to other perturbations; for example, MOnly-Num drops −61.24% on Range.Source: §5.2 · p. 6
5.3 False →False
- False-claim accuracy is in the mid-90s across types; Range causes drops across groups, while leave-one-out training counterintuitively gives the best Approximation accuracy (99.23%).MOnly-Random reaches 99.53% on Range.Source: §5.3 · p. 6
5.4 Cross-Dataset Generalisation
- On CLEF Spanish, MAll-Perturbs raises T→F accuracy from 73.71% to 97.07% (+23.36 pp), while T→T accuracy is essentially unchanged.The Spanish split is 95.8% False, so aggregate accuracy understates the gain on flipped cases; the paired comparison over 9,090 jointly parseable instances is significant (+2.33 pp, p = 8.0e−41).Source: §5.4 · p. 7, §5.5 · p. 7
5.5 Statistical Significance
- On AVeriTeC, the parseable-only baseline score exceeds MAll-Perturbs by 2.11 pp, a non-significant difference; the fine-tuned model instead has better verdict coverage.MBase returns no parseable verdict on 37.5% of instances, versus zero for MAll-Perturbs.Source: §5.4 · p. 7, §5.5 · p. 7
5.6 Sensitivity to Numerical Distance
- +7.50 pp within 10% relative numerical distance and +8.44 pp between 10% and 25% are the largest fine-tuning gains; the benefit falls to +2.91 pp beyond 100%.Every distance bin is individually significant, and the largest gains occur on near-misses rather than conspicuous edits.Source: §5.6 · p. 8
5.7 Evidence-Side Numerical Revisions
- 98.05% accuracy and 96.56% pair accuracy on VitaminC show the gains from evidence-side training; joint training reaches 98.25% on VitaminC and 98.20% on QuanTemp.Claim-side training alone scores 78.35% accuracy and 54.45% pair accuracy on VitaminC, below the untuned baseline; the two competencies are distinct but jointly learnable.Source: §5.7 · p. 8
6 Discussion
Adversarial fine-tuning repairs clean-data models’ collapse on label-flipping perturbations, but training composition affects performance on both perturbed and original claims. Results also show that robustness depends on perturbation type, model scale, adaptation method, and evaluation domain.
RQ1
- All-perturbation QLoRA models match or exceed their T→T performance on every T→F perturbation, unlike clean-data models that collapse to 5–13% accuracy.MOnly-Original is near-perfect on value-preserving evaluations but drops 83–90% on label-flipping perturbations.Source: §6 · p. 8
RQ2
- Single-perturbation models reach ≥96% on their own T→F type but collapse on at least one other, whereas random replacements can transfer across F→F.Distinct surface signals limit transfer between perturbation types; random-value replacement may teach a more general distrust of numbers.Source: §6 · pp. 8–9
Cross-corpus transfer
- Specialising on either VitaminC or QuanTemp costs 17–20 pp on the other, while joint training beats each specialist on its own test set.The gains are +0.68 pp on QuanTemp and +0.59 pp pair accuracy on VitaminC.Source: Table 6 · p. 9
RQ2
- Excluding Masked or Numeration degrades T→F performance below zero-shot, while excluding Approximation or Random has negligible effect.Masking adds a unique ### pattern, while Numeration requires digit-to-word equivalences.Source: §6 · p. 9
- Excluding Approximation yields 99.23% F→F accuracy but 53.96% T→T accuracy; excluding original examples recovers almost all T→T accuracy lost by all-perturbation training.The authors attribute the T→T shift to unperturbed examples rather than the perturbations themselves.Source: §6 · p. 9
RQ3
- The 0.6B all-perturbation model beats zero-shot 32B and frontier models on T→F by at least 14 pp, while accuracy reaches 98% at 4B and saturates at 8B.The scale-related jump comes mostly from Numeration, Range, and Approximation.Source: §6 · p. 9
- QLoRA models consistently outperform soft-prompt tuning; soft prompts remain below the zero-shot baseline despite improving over clean-data fine-tuning on T→F.The authors suggest prepended vectors may introduce conflicting biases rather than targeted numerical sensitivity.Source: §6 · p. 9
RQ4
- Adversarial fine-tuning transfers across languages, but on AVeriTeC its gain is in coverage rather than accuracy, and a false-prediction bias gives the baseline a narrow T→F edge.On Spanish CLEF 2025, the evidence is unstructured article text as in QuanTemp, so the shift is primarily lexical.Source: §6 · p. 9
7 Conclusion
The paper diagnoses brittleness to numerical claim perturbations and evaluates adversarial fine-tuning on counterfactual examples. The approach recovers accuracy on label-flipping inputs, with small QLoRA-tuned models outperforming larger zero-shot and frontier baselines on the hardest perturbations.
7 Conclusion
- Adversarial fine-tuning on counterfactual examples recovers near-perfect accuracy after clean-data models collapse to single digits on label-flipping inputs.The paper identifies Masked and Numeration as essential perturbation types, while Approximation and Random are redundant.Source: §7 · p. 9
- Small QLoRA-tuned models from 0.6B–8B surpass zero-shot 32B and frontier closed-source baselines on the hardest perturbations.The paper reports robustness transfer across languages and domains without target-language data.Source: §7 · p. 9
- The authors interpret cross-language and cross-domain transfer as evidence of a generalised numerical decision boundary rather than memorised edits.Source: §7 · p. 9
Limitations
The paper’s perturbations target scalar numerical expressions in claims, while its evaluation remains concentrated in English- and Spanish-language fact-checking. It also reports limits in model coverage, retrieval conditions, and mechanistic understanding.
Data and perturbation scope
- The perturbation generator rewrites claim-side scalar numbers, not evidence; jointly perturbing claim and evidence remains open.Robustness to unit conversions, date arithmetic, compound quantities, or numbers in tables is not guaranteed.Source: Limitations · p. 10
Model and evaluation setup
- Experiments use only Qwen3 models with QLoRA and soft prompts, and all assume gold evidence, leaving other model families and noisy retrieval untested.The soft-prompt baseline reuses QLoRA hyperparameters, making it a matched-budget rather than optimally tuned comparison.Source: Limitations · p. 10
Interpretability
- The paper lacks a mechanistic account of why adversarial fine-tuning works or which signals models use to detect numerical inconsistencies.The authors identify attribution, probing, and activation-patching analyses as needed to explain the gains and anticipate failure modes.Source: Limitations · p. 10
Ethical Considerations
The authors caution against deploying a specific model or configuration as an authoritative fact-checker, while noting both the potential misuse of perturbation techniques and the use of publicly available, non-sensitive data. The appendices report QuanTemp split counts and define the perturbation categories.
Ethical Considerations
- The authors do not endorse a model or configuration for real-world fact-checking deployment, and say outputs should remain decision support with human oversight.They note that perturbation techniques could help craft claims that evade automated fact-checkers; the data are publicly available and contain no personal or sensitive information.Source: Ethical Considerations · p. 10
A.1 QuanTemp Split Statistics
- The QuanTemp training split contains 931 original True claims and 2,434 original False claims; the validation split contains 324 True and 747 False claims.Source: Table 7 · p. 12, Table 8 · p. 12
A.2 Perturbation Type Definitions and Examples
- The perturbation taxonomy distinguishes value-preserving T1, value-changing T2, and random-replacement or masked-obfuscation T3 variants.Types marked T1 generate both value-preserving and value-changing variants; types marked T2 or T3 generate only that variant.Source: §A.2 · p. 12
A.3 Cross-Dataset Evaluation Sets
The appendix describes English and Spanish cross-dataset evaluation sets, training and prompting details, and parse behavior. Under relaxed parsing, many NCP outputs rejected by the strict JSON parser yield a usable label, raising reported NCP accuracy.
A.3 Cross-Dataset Evaluation Sets
- The English AVeriTeC 2.0 development set retains 119 numerical claim–evidence pairs; the Spanish CLEF 2025 set retains 2,633 claims and yields 16,460 instances including originals.AVeriTeC evidence is drafted as free-form passages from question–answer evidence; CLEF evidence comprises longer fact-check articles and is heavily skewed False.Source: §A.3 · pp. 12–13
B.1 Training Hyperparameters
- The QLoRA training hyperparameters are listed in Table 11, with additional SFTTrainer parameters left at their defaults.Source: §B.1 · p. 13, Table 11 · p. 13
B.2 Soft Prompt Tuning Formulation
- Soft prompt tuning prepends continuous prompt vectors to frozen input embeddings and updates only those vectors and the veracity head.The base parameters remain frozen, and the number of trainable prompt parameters is m·d, independent of model size; with m = 20, soft prompts are particularly weak on label-flipping perturbations.Source: §B.2 · pp. 13–14
B.3 Prompts
- The evaluation uses zero-shot instructions and also tests two-shot prompts and Numerical Counterexample Prompts with brief examples of false numerical claims.The two-shot prompt adds an instruction that two samples precede the task; the full NCP provides one example per perturbation type.Source: §B.3 · p. 14, Figure 4 · p. 14
B.4 Infrastructure and Hardware
- Experiments ran on a heterogeneous cluster with NVIDIA H100, A100, and Tesla V100 GPUs; GPU-hours are not reported because jobs were dynamically scheduled.Source: §B.4 · p. 14
C.1 Parse-Failure Rates
- Parse failures are concentrated in non-fine-tuned models and vary with prompting; Two-Shot reduces MBase failures below 3% across directions, while M32B Base stays at or below 4.40%.GPT-5.4 Pro parsing errors are 2.5–4%, and Gemini 2.5 Flash errors are 6.6–16.2%.Source: §C.1 · p. 14, Table 12 · p. 15
C.2 NCP Under Relaxed Parsing
- Relaxed parsing lowers NCP parse failures from 58.3% to 1.9% and raises overall NCP accuracy from 85.68% to 90.17%.It extracts the first label substring from a response; 96.8% of strict-parser rejections contain one.Source: §C.2 · p. 15, Table 13 · p. 15
- Relaxed parsing improves NCP results unevenly: recovered T→F and F→F cells improve, while T→T macro-average accuracy falls from 79.05% to 71.50%.The main tables retain strict-parser numbers for a consistent protocol; the appendix reports relaxed results for transparency.Source: §C.2 · p. 15, Table 13 · p. 15
D Statistical Significance
Paired tests qualify the cross-dataset findings: Spanish gains hold across protocols, while AVeriTeC shows a coverage advantage rather than a paired accuracy gain. Further analyses isolate Spanish label-flipping cases, quantify sensitivity to numerical distance, and describe the VitaminC evaluation design.
D Statistical Significance
- On CLEF Spanish, MAll-Perturbs gains +2.33 pp accuracy over MBase on the 9,090 jointly parseable instances (p = 8.0e−41); macro-F1 rises 8.43 pp.The gain is significant under all three protocols; macro-F1 is informative because the split is 95.8% False.Source: D Statistical Significance · p. 15, Table 14 · p. 16
E CLEF Spanish: Per-Perturbation Breakdown
- On Spanish T→F, MAll-Perturbs improves from 73.71% to 97.07% (+23.36 pp), while T→T performance is statistically indistinguishable and F→F remains near ceiling.The aggregate T2 column shows only +2.45 pp because it combines 553 T→F instances with 5,578 F→F instances; parse failures are also concentrated in hard cases, with MBase failing on 75.6% of T→F instances.Source: E CLEF Spanish: Per-Perturbation… · pp. 16–17, Table 16 · p. 17
F Sensitivity to Numerical Distance
- On QuanTemp, the benefit of adversarial fine-tuning is largest for near-miss values: +7.50 pp within 10% relative distance and +8.44 pp between 10% and 25%.It falls to +2.91 pp beyond 100%, and every distance bin is individually significant.Source: F Sensitivity to Numerical Distance · p. 17, Table 17 · p. 17
G.1 Subset Construction and Splits
- The VitaminC test subset contains 11,887 instances over 3,701 revisions, including 5,227 contrastive pairs; splits are disjoint by revision, claim text, and Wikipedia page.The subset retains real numerical revisions with binary SUPPORTS/REFUTES labels; the training sample is drawn by whole revision to keep contrastive pairs intact.Source: §G.1 · p. 18, §G.3 · p. 18
G.3 Training Details
- VitaminC evidence is a single revised sentence with a median of 26 words, unlike QuanTemp’s 535-word median; this tests numerical comparison under a short-evidence distribution shift.The reported test set has 11,887 instances and 5,227 contrastive pairs; the training sample comprises 16,000 instances drawn by whole revisions.Source: §G.1 · p. 18, §G.3 · p. 18
G.2 Pair Accuracy
- Pair accuracy counts a contrastive pair only when both evidence snapshots for a claim receive correct labels; errors are split into wrongful refutations and confirmations.This captures label-flipping performance and distinguishes the two error directions.Source: §G.2 · p. 18, Table 18 · p. 19
G.4 Additional Results
On VitaminC, prompting improves the fine-tuned model but does not close its gap to MBase, while the mixed training model modestly outperforms specialists on their respective test sets. Unquantised experiments reproduce the main qualitative findings across evaluation directions.
G.4 Additional Results
- TCP transfers to VitaminC test with +6.81 pp accuracy and +14.71 pp pair accuracy, yet remains 15.09 pp below MBase in pair accuracy.SCoT lowers pair accuracy to 46.15%; NCP is approximately neutral at 53.74%.Source: §G.4 · p. 19
H.1 True →False
- In unquantised T→F results, All Perturb Group is more robust than Base Group, while single-type training reaches perfect accuracy on its trained perturbation but collapses elsewhere.Exclusion Group achieves the best overall accuracies; Masked remains essential, while Approximation and Range signals are recoverable from the remaining training mix.Source: §H.1 · p. 20
H.2 True →True
- In unquantised T→T evaluation, MBase reaches 96.69% micro-average, while All Perturb Group scores in the 83–91% range.This reproduces the direction seen in the quantised results, where all-perturbations training loses ground on label-preserving inputs.Source: §H.2 · p. 20
H.3 False →False
- In unquantised F→F evaluation, most models achieve 96–99% accuracy, and leave-one-out models show little degradation on held-out perturbation types.Source: §H.3 · p. 20
H Unquantised Model Results
- Unquantised results reproduce the quantised findings: adversarial fine-tuning mitigates label-flipping brittleness at a small cost to label-preserving accuracy.The qualitative pattern holds independently of quantisation.Source: §H.3 · p. 20, H Unquantised Model Results · p. 20