Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants

1 Introduction

• In Figure 1, Qwen3-14B answers C (300) correctly at low diacritic-removal intensity but selects D at high intensity.The probability of the correct answer changes before the prediction flips.Source: Figure 1 · p. 1, §1 · p. 1
• Systematic paired comparisons of apostrophe omission, diacritic removal, and keyboard typos across French tasks and nested intensities remain limited, as do joint evaluations of accuracy and reliability.Prior French benchmarks primarily evaluate original inputs, while existing robustness studies examine other noisy-text settings.Source: §1 · p. 2
• The benchmark contains 4,200 original–low–high triplets (12,600 inputs), spanning seven task types and three French writing variations.High-intensity variants retain low-intensity edits and add further changes.Source: §1 · p. 2
• Accuracy and predicted-probability quality worsen with intensity; calibration transfers, and combining it with restoration gives the lowest Brier score and NLL while preserving restored accuracy.Keyboard typos have the strongest effects at the predefined high-intensity settings.Source: §1 · p. 2

French NLP and Evaluation

• FLUE, FQuAD, and FrenchBench broaden French task coverage, but do not provide matched evaluations of controlled French surface variations.FrenchBench was introduced alongside CroissantLLM for French LLM evaluation (Faysse et al., 2025).Source: §2 · pp. 2–3

Surface Variation in French Writing

• Corpus and behavioral studies document apostrophe omission, diacritic variation, and typing errors, motivating these three perturbation types.The cited work covers French text-message normalization, online dating, and typing habits.Source: §2 · p. 3

Robustness and Reliability

• Prior robustness studies address natural noise, injected errors, or typographical errors, while the joint effects of French surface variations on accuracy and prediction reliability remain less clear.Related work also examines calibration under noisy inputs and input correction, but the combined benefits of calibration and restoration are a focus here.Source: §2 · p. 3

3.1 Data Sources and Selection

• Seven French task types supply well-defined candidate answers; the benchmark selects 200 original questions per dataset and perturbation factor, yielding 4,200 triplets and 12,600 inputs.Question sets are non-overlapping across perturbation factors.Source: §3.1 · p. 3

3.2 Perturbation Construction

• Apostrophe omission, diacritic removal, and AZERTY-neighbor keyboard typos are applied separately, with high variants retaining all low edits and adding at least one modified word.Target proportions are 2.5% and 5% for apostrophe omission, and 5% and 10% for the other two types; samples unable to meet strict nesting are excluded.Source: §3.2 · pp. 3–4

3.3 Quality Validation

• Two models screen triplets for task-relevant information and answer validity; human review and a stratified audit check samples, with final accepted ratings reaching Krippendorff’s α = 1.000.The audit covers 210 triplets, or 5% of the benchmark.Source: §3.3 · p. 4

3.4 Dataset Splits and Statistics

• The 4,200 triplets are divided into 840 held-out and 3,360 test triplets, keeping each original and its variants together and splitting shared-source items by source group.Calibration temperatures are fitted exclusively on the held-out set.Source: §3.4 · p. 5

Models and Evaluation Setup

• Six zero-shot models from the Qwen3 and Gemma-3 families, spanning 4B to 32B parameters, receive the same French instructions and candidate options across input conditions.Candidate probabilities are computed by applying softmax to average token log-probabilities for valid answers.Source: §4 · p. 5

Evaluation Protocol and Metrics

• Accuracy measures answer correctness, Brier score and NLL assess predicted-probability quality, and ECE measures calibration using 10 equal-width confidence bins.Metrics are reported separately for original, low, and high inputs on 3,360 test triplets.Source: §4 · p. 5
• Temperatures fitted by minimizing NLL on held-out original inputs are transferred to perturbed tests; restoration accepts LanguageTool suggestions that follow the specified edit rules.Paired bootstrap tests resample source groups for main-table comparisons, with confidence intervals also reported for perturbation and intervention effects.Source: §4 · p. 5

5.1 Task Performance and Prediction Reliability

• Larger models in both families lose less accuracy under high intensity, but their Brier score and NLL increases do not consistently shrink with scale.Qwen3-32B loses less accuracy than Qwen3-14B but has a larger NLL increase; task-level declines also differ across models.Source: §5.1 · pp. 5–6
• Accuracy falls as perturbation intensity rises, with high-intensity losses of 0.63 to 1.79 percentage points; Brier score and NLL also rise monotonically across models.ECE increases over original inputs in 9 of 12 model–intensity conditions but has no consistent intensity trend.Source: §5.1 · p. 5, Table 4 · p. 6
• At predefined high intensity, keyboard typos cause the largest accuracy drops across all six models, from 0.89 to 4.02 percentage points, and the largest Brier score and NLL increases.Intensity does not always produce a larger accuracy loss for every model: Gemma-3-27B-it loses less accuracy under high- than low-intensity typos even as its NLL rises.Source: §5.1 · p. 6

5.2 Prediction Flips and Confidence Structure

• Across all six models, originally correct predictions in the smallest-margin quartile have the highest flip rates at both intensities, and rates rise from low to high intensity.Higher-margin quartiles have substantially lower flip rates, despite relatively modest overall accuracy declines.Source: §5.2 · pp. 6–7, Figure 4 · p. 7
• Confidence–accuracy curves stay close from 0.8 to 1.0 confidence, but some lower-confidence bins differ across input conditions; fitted calibration temperatures remain similar across conditions.For Qwen3-14B in the 0.4–0.5 confidence bin, mean confidence is near 46% for both original and high-intensity inputs, while accuracy is 50.91% and 33.58%, respectively.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 Calibration Transfer and Input Restoration

• Calibration fitted on original inputs reduces ECE, Brier score, and NLL for all six models at both intensities without changing accuracy.The reductions vary by model, with particularly large ECE and Brier score reductions for Gemma-3 models.Source: §5.3 · p. 8, Table 5 · p. 8
• Restoration improves accuracy for five of six models at low intensity and all six at high intensity; adding original-input calibration preserves that accuracy and yields lower Brier score and NLL than either intervention alone across all 12 conditions.ECE does not consistently favor the combined approach.Source: §5.3 · p. 8

• Accuracy decreases and Brier score and NLL increase with perturbation intensity across six models.Source: §6 · p. 8
• At high intensity, larger models show smaller accuracy losses, but their Brier score and NLL do not improve consistently with scale.Source: §6 · p. 8
• At predefined high-intensity settings, keyboard typos cause greater accuracy and predicted-probability degradation than the other variations.Source: §6 · p. 8
• Calibration fitted on original inputs transfers to variants and, combined with restoration, improves Brier score and NLL beyond either method alone while retaining restoration’s accuracy gains.Source: §6 · p. 8

• The benchmark applies apostrophe omission, diacritic removal, and keyboard typos separately at two predefined perturbation intensities, leaving their joint effects untested.Source: Limitations · p. 9
• Its seven task types come from existing French datasets that do not cover all French-use contexts; broader domains and text types remain possible extensions.Source: Limitations · p. 9

Ethical Considerations

• The benchmark is intended for research on model performance and prediction reliability, and the studied writing variations should not be used to infer language proficiency or social identity.The authors encourage using the benchmark to improve support for diverse writing practices.Source: Ethical Considerations · p. 9

A Benchmark Construction and Quality Review

• The benchmark contains 4,200 original–low–high triplets, or 12,600 inputs, with 840 held-out and 3,360 test triplets.Source: A Benchmark Construction and Quality… · p. 12

A.1 Perturbation Rules

• Apostrophe and diacritic edits exclude specified lexical collisions, while keyboard typos change one non-initial letter to an AZERTY neighbor and exclude common-word substitutions.Both variants share an edit plan, with all low-intensity edits retained in the high-intensity version.Source: §A.1 · p. 12

A.2 Automatic Task-Impact Review

• Two models independently screened candidate triplets using shared instructions and decision rules, assessing each variant for task-relevant changes, reference conflicts, out-of-scope edits, and structural damage.Source: §A.2 · p. 12
• A triplet was flagged when either variant failed; failed or invalid Gemini assessments triggered targeted human review, and replacements required valid pass judgments from both models.The team judged the DeepSeek flag to be a false positive.Source: §A.2 · p. 12

A.3 Human Review Criteria and Outcomes

• Human reviewers judged variants separately against the original and reference answer, failing clear task-relevant changes, unreadable passages, structural loss, or out-of-scope edits.Four triplets were replaced after majority judgments or a source-question check, retaining their dataset, perturbation factor, and split assignments.Source: §A.3 · pp. 12–13
• 210 triplets were randomly audited across 420 variants; after guideline clarification, accepted ratings reached Krippendorff’s α of 1.000 and unanimous agreement of 100.00%.Initial values were α = 0.969 and 98.33% unanimous agreement.Source: §A.3 · pp. 12–13

B.1 Implementation and Hardware

• Six Qwen3 and Gemma-3 checkpoints ran on specified NVIDIA GPUs using vLLM, while DeepSeek and Gemini API models performed automatic quality screening.The GPU environments used vLLM 0.19.1, PyTorch 2.10.0, CUDA 12.8, and Transformers 5.6.2.Source: §B.1 · p. 13

B.2 Prompt Template

• The shared French prompt uses task-specific instructions and valid answer options, with model-specific system or instruction headers and a response prefix.The task instructions cover seven multiple-choice task types.Source: §B.2 · p. 13

B.3 Candidate Scoring and Runtime

• Predictions use mean token log probabilities for each valid answer code, normalized across candidates, without chain-of-thought generation.Original, perturbed, and restored inputs use the same scoring procedure.Source: §B.3 · p. 13

B.4 Input Restoration

• LanguageTool suggestions restore apostrophes, diacritics, or AZERTY-neighbor typos, and accepted non-overlapping edits are applied before scoring each input condition.The same restoration rules are applied separately to original, low-intensity, and high-intensity inputs.Source: §B.4 · pp. 13–14

B.5 Temperature Fitting and Metrics

• Each model’s temperature is fitted by minimizing NLL on 840 held-out original triplets, then fixed for evaluation; accuracy, Brier score, NLL, ECE, and prediction margins are defined.Uncalibrated probabilities use T = 1; condition-specific temperatures are fitted separately on low- or high-intensity held-out variants.Source: §B.5 · p. 14

B.6 Paired Statistical Comparisons

• Paired bootstrap comparisons resample shared source groups, recompute metric differences, and apply Holm adjustment across models, intensities, and metrics.The procedure uses 10,000 resamples; intervention intervals keep temperatures fixed at their original held-out estimates.Source: §B.6 · p. 14

C Detailed Perturbation Results

• The uncalibrated results tables report model-, dataset-, and perturbation-factor-level values for Original, Low, and High conditions.Accuracy and ECE are percentages; Brier score and NLL retain their original scales.Source: C Detailed Perturbation Results · p. 14, Table 7 · pp. 15–17

C.1 Paired Prediction Changes

• Paired prediction-change results distinguish all flips from harmful and helpful flips, and report high-confidence errors using a confidence threshold of 0.8.Table 8 reports variant-minus-original changes in flip and HCE rates, accuracy, mean confidence, and correct-answer probability.Source: Table 8 · p. 18, §C.1 · p. 18

D Prediction Flips and Calibration Temperatures

• The flip analysis groups originally correct predictions by original top-two probability margin, reports Q1’s share of harmful flips, and examines condition-fitted temperatures and deletion-control margins.The deletion control removes words modified by a variant; Variant minus Deletion estimates their contribution in modified form to the correct-candidate margin.Source: D Prediction Flips and Calibration… · p. 18, Table 9 · p. 18, Table 10 · p. 18

E Calibration and Restoration Results

• The calibration and restoration appendix supplements the main analysis with task-level calibration findings and original-input controls.The same restoration procedure is applied to original test inputs as a control.Source: E Calibration and Restoration Results · p. 18, §E.2 · p. 19

E.2 Input Restoration and Original-Input Control

• Restoration results report accuracy changes and repair and harm counts for 3,360 test triplets, with each count referenced to the pre-restoration prediction.Calibration and Combined use the original fitted temperature; Combined applies it after restoration, without fitting a temperature on restored inputs.Source: Table 12 · p. 19, §E.2 · p. 19, §E.1 · p. 19

E.1 Calibration Transfer

• Global calibration improves pooled NLL, but benefits vary by dataset; dataset-specific original-input temperatures reverse most dataset-level NLL increases at both perturbation intensities.The appendix also reports task-level calibration findings and original-input controls.Source: E Calibration and Restoration Results · p. 18, §E.1 · p. 19

F Paired Effect Estimates

• Paired effect-estimate tables report metric differences and pointwise 95% bootstrap confidence intervals for perturbation and intervention comparisons.Negative probability-score differences and positive accuracy differences indicate improvement; calibrated scores use the original held-out temperature.Source: F Paired Effect Estimates · p. 19, Table 13 · p. 20, Table 14 · p. 21