Source-linked AI summary
Text-Only LLM Correctors Break Correct Transcripts: A Repair-Damage Analysis of Audio Grounding in ASR Error Correction
Vasiliy Kudryavtsev, Kirill Borodin, Nikita Vasiliev, Mikhail Gorodnichev, Grach Mkrtchian
TL;DR
Aggregate WER mixes words that LLM correctors repair with words they damage, leaving unclear what source audio changes. The paper compares text-only and audio-grounded correction for model-generated drafts and independent recogniser outputs. With Gemini 2.5 Flash, grounding reduces damage in both settings, while repair effects differ by hypothesis source; the prompts also differ, so the contrasts do not isolate audio’s effect.
Problem
Aggregate WER combines repairs and damage, and prior multimodal comparisons leave unclear what source audio changes in correction.
Source: §1 · p. 1Method
The study compares text-only and audio-grounded correction for a model’s draft and an independent recogniser’s output.
Source: §2.1 · p. 1Results
Takeaways & Limitations
For an over-correcting system, the authors recommend reporting break-rate and the three components alongside WER, and spending the audio call on an independent recogniser’s hypothesis.
Source: §6 · p. 4The audio contrasts do not isolate audio’s effect because instructions change between arms, and only one model carries the full factorial.
Source: §5 · p. 4
Abstract
from arXiv · showhide
LLM correctors of ASR output are judged by aggregate WER, which hides what they fix and what they break. We decompose every WER change exactly into repaired tokens, damaged tokens and insertions, and use this to ask what the source audio adds to a corrector. We compare correction with and without the audio, for the model's own draft and for the output of an independent recogniser, on three corpora. With Gemini 2.5 Flash, text-only correction of its own draft breaks 13-32% of already-correct transcripts; with the audio, 1-3%. Damage falls in both settings. Repair falls against the own draft and rises against the recogniser. On medical dictation, kept insertions cancel the gain in net WER. The open-weight Voxtral, which rarely edits, does not show this pattern. Prompts differ between the two arms, so the contrasts bound the effect of the audio and do not isolate it.
1. INTRODUCTION
Aggregate error rates mix words repaired with words damaged, leaving unclear what source audio changes in LLM correction. The paper’s accounting and comparisons find that grounding reduces damage, while repair effects differ by hypothesis source.
1. INTRODUCTION
- Aggregate error rates mix repairs and damage, so the paper asks what changes when a corrector can hear the audio.Prior work has compared multimodal correction settings, but the supplied passage identifies the change behind their aggregate results as unaccounted for.Source: §1 · p. 1
- Text-only correction breaks 13–32% of already-correct transcripts; grounding removes most damage for both the model’s draft and an independent recogniser’s output.Repair effects have opposite signs across those two settings, and insertions hide the benefit of grounding in net WER on medical dictation.Source: §1 · p. 1
- The authors interpret grounding as reducing damage for an over-correcting system and adding repair for an under-correcting one, but call this a hypothesis based on two models.A near-ceiling synthetic corpus shrinks the repair effect by an order of magnitude, while an open-weight model that rarely edits shows no consistent damage drop.Source: §1 · p. 1
- The contribution is word-level accounting, a factorial comparison, and the resulting findings about audio-grounded correction.The authors state that the accounting is exact with respect to WER.Source: §1 · p. 1
2. METHOD
The method compares correction with and without audio for both a model’s own draft and an independent recogniser’s output. It decomposes WER changes into repair, damage, and insertion components, while noting that the prompt wording also changes between conditions.
2.1. Conditions
- The five conditions compare audio-grounded and text-only correction of the model’s draft and an independent recogniser’s output, plus an audio-only control.Each output is scored against the system that supplied its input; within each setting the clips and hypothesis stay fixed, but prompt wording changes.Source: §2.1 · p. 1
2.2. Repair, damage and insertions
- The method decomposes WER change into repair, damage, and insertion components, and defines break-rate and fix-rate for error-free and erroneous inputs.The accounting identity has blind spots: wrong-token substitutions are not counted, and component rates depend on the alignment path; an alternative backtrace shifted rates by at most 0.02 pt.Source: §2.2 · pp. 1–2
2.3. Prompts
- Grounded prompts ask for transcription using the hypothesis as a hint or phonetic guide, while text-only prompts ask for editing; thus the two arms differ in instructions as well as audio.This makes the damage contrast conservative, while the cross-system repair effect is an upper bound on the audio contribution; control B’s prompt alone moves medical WER by 2.53 pt.Source: §2.3 · p. 2
3. EXPERIMENTAL SETUP
The experiment compares correction across three corpora and multiple systems, with outputs scored against their input systems. It also reports token-flow measures and documents preprocessing, interval estimation, and exclusions.
3. EXPERIMENTAL SETUP
- The study uses 3,000 synthetic Russian utterances, 300 Russian audiobook utterances, and 300 English medical-dictation utterances; language is confounded with corpus and recogniser.Requiring all seven outputs to be non-empty leaves n = 2,260, 270, and 284, respectively.Source: §3 · p. 2
- Gemini 2.5 Flash is the primary corrector; the study also runs Voxtral-Small-24B across all cells and GPT Audio on cell F.The recognisers are GigaAM for Russian and Whisper Large-v3-Turbo for English.Source: §3 · p. 2
- Corpus-level WER uses normalized, lower-cased text with Cyrillic ё folded to е and punctuation stripped, without inverse text normalization.Source: §3 · p. 2
- Table 1 reports input-system WER and each cell’s ΔWER against its input; Table 2 contrasts grounded-minus-text-only repair, damage, insertions, and net change.Source: Table 1 · p. 2, Table 2 · p. 2
- Reported 95% paired utterance-level bootstrap intervals use 20,000 resamples, are unadjusted for multiplicity, and ignore speaker clustering.The authors rely on sign consistency across corpora rather than single borderline intervals.Source: §3 · p. 2
- For Gemini, missing outputs are excluded; synthetic cell T lacks 738 outputs and F lacks 269, and the authors did not verify domain balance in the common subset.On natural corpora, 30 and 16 utterances drop out, almost all because the draft is empty; excluding these failures flatters draft WER.Source: §3 · p. 2
4. RESULTS
Grounded correction beats text-only correction on every corpus when correcting an independent recogniser, while self-correction shows no detectable net difference on natural speech. Decomposing WER reveals different repair and damage patterns, with insertions, corpus ceiling effects, and model behavior qualifying the results.
4.1. Net effect
- Grounded correction beats text-only correction on every corpus in the cross-system setting; on natural speech, self-correction shows no detectable net difference.Self-correction improves by +2.92 pt on the synthetic corpus, but the natural-corpus intervals include zero.Source: §4.1 · pp. 2–3
4.2. Repair, damage and break-rate
- In self-correction, grounding lowers repair by 0.15–0.74 pt and damage by 0.88–2.83 pt; against ASR, repair rises by +0.09, +0.95, and +1.25 pt.For error-free drafts, text-only break-rates are 31.9%, 16.3%, and 12.8%, versus 2.6%, 1.0%, and 2.3% with audio.Source: §4.2 · p. 3
4.3. Insertions
- On medical dictation, grounded correction leaves insertions nearly unchanged at 4.37% of reference tokens, so their −1.08 pt component cancels +0.98 pt from repair and damage.Split compounds and fillers contribute to insertions, and spelling conventions can affect all three components; compound- and filler-insensitive scoring could flip the net A−C contrast.Source: §4.3 · p. 3
4.4. Robustness and the synthetic-corpus ceiling
- The synthetic corpus is near-ceiling, with 65% of ASR outputs and 79% of drafts already correct; three conclusions there do not hold on natural speech.Its cross-system repair effect is +0.09 pt, against +0.95 and +1.25 pt on the natural corpora.Source: §4.4 · p. 3
4.5. Domain terminology, per-domain results and noise
- On the synthetic corpus, grounded correction keeps domain-term recall at 69.3% versus 66.1% after text-only self-correction, while text-only correction raises draft WER in all ten domains.Grounded correction also raises cross-system domain-term recall by +4.2 pt, against +2.0 pt for text-only correction.Source: §4.5 · p. 3
4.6. Other models
- Voxtral does not replicate Gemini’s pattern: grounding raises repair in all six contrasts, while damage moves inconsistently and insertions make net A−C negative on two corpora.Voxtral text-only self-correction rarely edits, with a 1.5–3.8% fix-rate; GPT Audio degrades two of three corpora in cell F, without intervals.Source: §4.6 · p. 4
5. DISCUSSION AND LIMITATIONS
The authors offer a tentative explanation for why grounding’s repair effect differs between self-correction and cross-system correction, while emphasizing that the comparison does not establish who wrote the hypothesis as the cause. Their practical conclusions and limitations are bounded by the tested model, prompts, and corpora.
5. DISCUSSION AND LIMITATIONS
- Table 5 covers Voxtral-Small-24B on subsets of the same clips; its insertion rows are derived by subtraction and have no intervals.Bold entries indicate 95% intervals excluding zero.Source: Table 5 · p. 4
- The sign difference holds between the tested settings, but does not establish an effect of who wrote the hypothesis; Voxtral contradicts that interpretation.The proposed explanation—that audio adds new evidence for an independent ASR hypothesis but not for the model’s own draft—is explicitly a hypothesis.Source: §5 · p. 4
- The draft may anchor self-correction to its own errors: on medical dictation, the no-draft control B beats A (+2.53 against +0.79 pt), though the draft explanation remains uncertain.The data cannot exclude cell A behaving as a near-deterministic retranscription that ignores the draft.Source: §5 · p. 4
- On this model, extra repair from audio appeared only with an independent hypothesis; in self-correction on natural speech, its measurable benefit was fewer broken utterances, not lower WER.The authors caution that none of these findings transfers to a model that rarely edits.Source: §5 · p. 4
- Prompt wording changes with audio, so only a matched-prompt design would isolate the audio; only one model has the full factorial, and two audio LLMs are commercial APIs subject to version drift.The synthetic corpus derives from Wikipedia text the LLMs have probably seen in pretraining, and the study does not compare against a task-trained corrector.Source: §5 · p. 4
6. CONCLUSION
Aggregate WER hides both the damage done by text-only LLM correction and what the audio changes. The conclusion recommends reporting break-rate and the three WER components alongside WER.
6. CONCLUSION
- Report break-rate and the three WER components alongside WER; for an over-correcting corrector, the authors recommend spending the audio call on hypotheses from an independent recogniser.Source: §6 · p. 4
8. COMPLIANCE WITH ETHICAL STANDARDS
The study uses publicly released speech corpora and collected no new human-subject data.
8. COMPLIANCE WITH ETHICAL STANDARDS
- The numerical study uses publicly released speech corpora, including a public medical ASR evaluation set with no protected health information; no new human-subject data was collected, so ethical approval was not required.Source: §8 · p. 5