Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

1 Introduction

• ASR models have less training text than LLMs, leaving a semantic gap that matters especially for named entities and uncommon words.The passage gives Whisper-large-v3’s approximately 5 million hours of speech as an example of the scale disparity.Source: §1 · p. 1
• ASR-LLMs integrate pretrained LLM backbones with speech encoders and can outperform systems trained from scratch or corrected after recognition.The cited performance pattern suggests semantic knowledge transfers more effectively through warm initialization than post-hoc correction.Source: §1 · pp. 1–2
• Cosine similarity and relative magnitude between ASR-LLM and base-LLM hidden states provide signals for identifying tokens with high semantic dependence.The analysis compares the adapted model’s representations with those of its original base LLM.Source: §1 · p. 2
• Hybrid Search selects tokens with features associated with high semantic dependence, then applies targeted correction using the original base LLM.The framework uses hidden-state interaction features to govern whether LLM log-probabilities are incorporated into prediction.Source: §1 · p. 2, Figure 1 · p. 2
• Hybrid Search improves named-entity recognition over greedy search, beam search, rescoring, and late fusion while maintaining WER comparable to beam search.The paper concludes that an ASR-LLM can further leverage its base LLM at inference time after warm initialization.Source: §1 · p. 2

2 Background

• The study examines self-correction in LoRA-based ASR-LLMs, using Phi-4-Multimodal as a case study and Granite-4.0-1B-Speech to validate generality.Both models retain functional LLM backbones and have been trained on substantial speech data, from 100k to 2.3 million hours.Source: §2 · pp. 2–3
• Prior work reports warm-initialized ASR-LLMs outperforming models trained from scratch and post-hoc LLM-corrected variants on standard ASR benchmarks.The background notes fully finetuned and parameter-efficient adaptation paradigms.Source: §2 · p. 2
• The study reports NE-ER alongside WER because WER places limited emphasis on semantically consequential errors such as named-entity errors.Prior approaches cited include domain-specialized language models, explicit domain tags, embedded NER modules, and LLM-based similarity scores.Source: §2 · p. 3

3.1 Extraction of Hidden State Interactions

• Angle and magnitude features compare ASR-LLM and base-LLM hidden states at corresponding layers, using matched references to measure ASR-induced representational change.The ASR-LLM states are speech-conditioned, while the base-LLM states are computed from the corresponding text-only context.Source: §3.1 · p. 3

3.2 How Hidden State Interactions Reveal Token Semantic Dependence

• Named entities serve as a semantic-dependence proxy, while non-named entities serve as a proxy for tokens grounded more strongly in acoustic evidence.The analysis uses YODAS and Phi-4-Multimodal with Phi-4-Mini as its base LLM.Source: §3.2 · p. 4
• In layers 0–7, both features stay near 1 with small token-level variation, leaving little room to distinguish token categories by their trajectories.Source: §3.2 · p. 5
• Intermediate-layer peak features distinguish named-entity trajectories, with the largest distributional discrepancy driven primarily by continuation tokens in multi-token words.The peak features are top-three means from layers 10–28; named entities occupy a high-AngleFeat, high-MagFeat region.Source: §3.2 · p. 5
• Layer 31 shows a weaker named-entity shift toward lower AngleFeat and higher MagFeat, so the method does not use this late-layer effect.Source: §3.2 · p. 5

4.1 Targeted Self-Correction

• Global LLM probability fusion can correct semantically dependent predictions but is susceptible to hallucinations when the decoded token is acoustically dominant.Source: §4.1 · p. 5
• The selector targets regions where named-entity density exceeds non-named-entity density, using the two-dimensional AngleFeat and MagFeat representation.For multi-token words, selection uses continuation tokens because starting tokens do not show clear separation; selecting one continuation expands selection to the full word.Source: §4.1 · p. 6

4.2 Self-Correction Algorithm

• Table 1 characterizes targeted decoding as an efficient beam-search alternative, while the error-correction variant takes a second pass and further improves on beam search.Source: Table 1 · p. 6
• Hybrid Search mixes greedy and beam search, expanding only targeted words to avoid unstable comparisons between targeted and non-targeted token scores.Source: §4.2 · p. 6
• After beam expansion, decoding returns to greedy search once beams converge on the same non-targeted token, rolling back any decoding beyond that token.Figure 3 illustrates this transition and the backtracking to a beginning-of-word token.Source: §4.2 · p. 6, Figure 3 · p. 6

Experimental design

• The study evaluates one-pass Hybrid Search for Decoding and two-pass Hybrid Search for Error Correction, which refines the best beam-search hypothesis.For non-targeted positions in the correction setting, decoding uses teacher forcing on the beam-search hypothesis.Source: §5 · p. 7

Evaluation data and metrics

• Evaluation uses Phi-4-Multimodal on seven Open ASR Leaderboard subsets and adds Common Voice to assess robustness across diverse speakers.The seven subsets are AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, TED-LIUM, and VoxPopuli.Source: §5 · p. 7
• The benchmark reports WER and named-entity error rate, using beam size B = 5, interpolation weight α = 0.2, and τ = 1/2 by default.The error-correction setting is also evaluated with τ = 1.Source: §5 · p. 7

6.1 Baselines

• Phi-4-Multimodal beam search achieves the best WER on 5 of 7 benchmarks and outperforms the ensemble of prior corrective methods overall.The comparison covers prior methods that also incorporate beam search within their correction loops.Source: §6.1 · p. 7

6.2 Hybrid Search for Decoding

• Hybrid Search recovers 96.4% of beam search’s named-entity improvement over greedy search, while recovering 25% of its WER improvement.It targets 10% of tokens on average; with B = 5, its amortized beam size is 1.4, or approximately 1.4× greedy-decoding cost.Source: §6.2 · p. 7
• Removing LLM correction leaves WER comparable but reduces named-entity improvement to 68.3% of beam search’s gain, down from 96.4%.The authors suggest targeted regions benefit from both additional search and base-LLM linguistic information.Source: §6.2 · p. 7

6.3 Hybrid Search for Error Correction

• Average NE-ER falls by 3.3% with Hybrid Search over beam search while WER remains comparable, whereas rescoring and late fusion noticeably degrade WER.Table 2 reports the benchmark comparison; under τ = 1/2, NE-ER decreases from 18.29% to 17.69%.Source: Table 2 · p. 8, §6.3 · p. 8
• Hybrid Search’s largest reported relative NE-ER gains occur on TED-LIUM (12.7%) and SPGISpeech (8.8%), with significant tests reported for both NE-ER and WER.The two-sided exact McNemar p-values are 3.56×10^-38 for NE-ER and 9.19 × 10^-5 for WER.Source: §6.3 · p. 8

7.1 Analysis of Correction Examples

• Most Hybrid Search edits involve semantically specific expressions, particularly named entities; foreign and uncommon lexical items also frequently appear among edited tokens.The authors report that hallucination-inducing edits are minimal, and that most edits occur within multi-token words.Source: §7.1 · pp. 8–9

7.2 Is Targeted Correction Necessary?

• Hybrid Search substantially improves WER over rescoring while maintaining NE-ER, supporting targeted correction over untargeted refinement.Figure 4 compares relative WER and NE-ER improvements across test sets against beam search.Source: §7.2 · p. 9, Figure 4 · p. 9

7.3 Are Intermediate Interactions Necessary?

• Targeting multi-token words by surface form alone fails to preserve beam search’s WER robustness, indicating hidden-state interactions are essential for effective targeting.The τ = 0 baseline targets only multi-token words and omits intermediate features.Source: §7.3 · p. 9

7.4 Complexity Analysis

• Feature extraction costs about 10^6 FLOPs per decoding position versus about 10^9 for a model forward pass, and selective expansion yields 1.4× greedy-decoding cost.The analysis states ASR-LLM and internal LLM execution can run in parallel, keeping latency bounded by the same N autoregressive decoding steps.Source: §7.4 · p. 9

• Hybrid Search uses cosine similarity and relative norm between ASR-LLM and base-LLM hidden states to identify distributional signals associated with semantically dependent tokens.The hidden-state observations are reported for Phi-4-Multimodal and Granite-4.0-1B-Speech.Source: §8 · p. 9
• Experiments with Phi-4-Multimodal show improved NE-ER while maintaining competitive WER, without external models or additional training.The authors conclude that residual base-LLM knowledge may remain exploitable at inference time even after warm initialization.Source: §8 · p. 9

Limitations

• End-to-end Hybrid Search is evaluated only on Phi-4-Multimodal, so generalization of the full correction method across ASR-LLM architectures remains unestablished.Feature distributions, layer aggregation, and thresholds may depend on the model; other domains such as singing ASR may require calibration or adaptation.Source: Limitations · p. 10

A Extended Related Work

• Speech foundation models such as Whisper, OWSM, and Canary demonstrate multilingual, multi-task, or cross-domain speech capabilities at large training scales (Radford et al., 2023; Peng et al., 2025; Puvvada et al., 2024).Source: A Extended Related Work · p. 13
• This work adopts Phi-4-Multimodal as its ASR–LLM backbone; such multimodal LLMs often use parameter-efficient adaptation, including LoRA, to preserve pretrained knowledge while maintaining training efficiency (Abouelenin et al., 2025).Source: A Extended Related Work · p. 13
• Prior generative error correction often uses sequential two-pass inference, while synchronous fusion methods still require parallel inference from separate ASR and LLM models (Chen et al., 2023; Radhakrishnan et al., 2023; Hsu et al., 2025a; Mittal et al., 2024; Hori et al., 2025).The paper contrasts these approaches with its unified ASR-LLM architecture, which uses internal hidden states during a single forward pass.Source: A Extended Related Work · p. 13
• Prior hidden-state research treats layer-wise changes as latent reasoning trajectories and uses geometric properties to estimate correctness or semantic uncertainty (Chen et al., 2025; Wang et al., 2025).Source: A Extended Related Work · p. 14

B.1 Late Layer Feature Discovery

• Named-entity tokens shift toward lower cosine and higher norm, with the shift most pronounced at the starts of multi-token words.Source: §B.1 · p. 14
• Starting tokens of multi-token names may be uncertain because acoustically similar candidates and spelling variations need longer context to resolve.Source: §B.1 · p. 14
• A balanced probability mixture over competing candidates requires greater departure from the original LLM embedding space, which emphasizes semantic relations over acoustic similarity.Source: §B.1 · p. 14

B.2 Hybrid Search Algorithm

• Hybrid Search alternates between greedy-search and beam-search states; “advance one decoding step” covers forward progress in either state.Source: §B.2 · p. 14

C.1 Generalization of targeted approaches

• Granite-4.0-1B-Speech shows similar distribution shifts in AngleFeatinter and MagFeatinter.Source: §C.1 · p. 14

C.2 Generalization of targeted region

• A targeted region derived from YODAS shows consistent concentration effects in named-entity tokens when applied to other Open ASR Leaderboard datasets.Source: §C.2 · pp. 14–15

C.3 Case Examples of Correction

• Significant Hybrid Search edits mostly involve named entities; changes to acoustically dominant tokens that risk hallucination are relatively uncommon.The examples include both edits that match the gold transcript and edits that do not.Source: §C.3 · p. 15
• Hybrid Search changes “ventran” to “van tran,” matching the gold transcript’s rendering of the name.The beam-search transcript has “ventran”; the Hybrid Search output and gold transcript both have “van tran.”Source: §C.3 · p. 15
• In another example, Hybrid Search changes “climbed” to “clambered,” differing from the gold transcript.The beam-search transcript and gold both say “climbed up the building,” while the Hybrid Search output says “clambered up the building.”Source: §C.3 · p. 15
• The correction examples compare Beam Search, Hybrid Search τ = 1, and output 2 against Gold on business, financial, and medical utterances.For example, the outputs differ on “actual results” and on wording about costs contained in Irapuato.Source: §C.3 · p. 19

[TED-LIUM]

• TED-LIUM examples expose transcription differences involving names and phrases, including Vinod Khosla, Jonas Salk, and Cedefop.The passages show Beam Search, Hybrid Search, output 2, and Gold transcripts for varied speech excerpts.Source: [TED-LIUM] · p. 20

C.4 Layer-wise Features Visualization

• Named-entity words such as “Mortimer”, “Adler”, and “Randi” show higher cosine similarities and norm ratios in intermediate layers.Figures 7 and 8 visualize AngleFeat and MagFeat heatmaps for the Common Voice example, with some named-entity tokens showing higher values.Source: §C.4 · p. 21, Figure 7 · p. 23, Figure 8 · p. 24