Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR

• Conversational context matters because prior turns can introduce entities, pronunciations, and discourse cues needed to interpret utterances.Source: §1 · p. 1
• Most ASR systems process utterances independently, often failing on rare contextual entities such as names, locations, and domain-specific terminology.Source: §1 · p. 1
• After supervised multi-turn training, conversational context improves LLM-based ASR, with the clearest gains on contextual entities measured by Bias-WER.Source: §1 · p. 1
• Abstract Compression replaces each prior-turn audio sequence with a fixed number of learned latent tokens while retaining the transcript explicitly.Source: §1 · p. 2, Figure 1 · p. 2
• Across in-domain and out-of-domain evaluation, the compressed representation recovers part of raw-context conditioning’s gains with a smaller prior-turn audio footprint.Source: §1 · p. 2
• The work contributes multimodal context analysis, Abstract Compression with two-stage training, and ablations of compression design choices.Source: §1 · p. 2

• Contextual ASR has used contact names, locations, and domain-specific phrases to improve recognition of rare words and named entities.Source: §2 · p. 2
• Recent multimodal ASR models combine pretrained LLMs with speech encoders or unified audio-text processing architectures.Source: §2 · pp. 2–3
• Most prior LLM-based ASR work emphasizes single-utterance recognition, instruction following, or general multimodal capability rather than multi-turn conversational context.Source: §2 · p. 3
• Efficient sequence-modeling research reduces long-context cost through pruning, pooling, summarization, memory tokens, and latent bottlenecks.Source: §2 · p. 3
• The paper studies richer multimodal conversation context for ASR and a task-driven bottleneck at the intersection of contextual ASR, multimodal LLMs, and efficient long-context modeling.Source: §2 · p. 3

• The experiments use PHI-4-MULTIMODAL’s speech-processing capabilities to isolate conversational context’s impact on speech recognition.Source: §3 · p. 3
• An audio encoder and projection module map waveform inputs into the LLM’s input embedding space.Source: §3 · p. 3
• Audio tokens are interleaved with text embeddings and processed by the PHI-4-MINI LLM.Source: §3 · p. 3
• In the single-turn baseline, the model receives the current audio with a standard chat-template transcription instruction.Source: §3 · p. 3

• A conversation is modeled as audio segments and transcripts across N turns, with the final turn transcribed using the preceding N−1 turns as context.Source: §4 · p. 3
• The inference-time setup prepends completed prior turns to the current transcription request using the model’s multi-turn chat format.Source: §4.1 · p. 4
• Inference-time raw-context prompting degraded performance relative to the single-turn baseline, indicating that prompting alone was insufficient for reliable cross-turn use.Source: §4.1 · p. 4
• Supervised multi-turn fine-tuning trains the model to predict the final transcript from prior turns and current audio.Source: §4.2 · p. 4
• Raw-context fine-tuning improved Bias-WER and recovery of contextual entities, but high-resolution prior-turn audio makes total prompt length grow rapidly with conversation length.Source: §4.2 · p. 4

• Raw-context audio becomes increasingly costly with conversation length because each prior turn contributes many audio tokens, whereas transcripts remain comparatively compact.Source: §5 · p. 4
• Abstract Compression maps each variable-length prior-turn audio sequence to K learned latent tokens and leaves the corresponding transcript explicit.Source: §5.1 · p. 4
• Turn-specific learnable queries cross-attend to each turn’s audio tokens, producing latent vectors that capture that turn’s context through a fixed-length information bottleneck.Source: §5.1 · p. 4
• Compression applies only to prior turns; the current utterance remains uncompressed to preserve its full acoustic detail.Source: §5.1 · pp. 4–5
• Stage 1 freezes the base model and trains the compression module for compressed-audio-to-LLM alignment, while Stage 2 jointly fine-tunes compression and audio LoRA parameters on compressed multi-turn inputs.Source: §5.2 · p. 5
• Stage 2 uses a context-length curriculum that starts without prior turns and progressively increases the maximum number of available turns.Source: §5.2 · p. 5

• DefinedAI is the main in-domain dataset, WoW is the out-of-domain evaluation set, and LibriSpeech 960h is used only for Stage 1 compression training.Source: §6.1 · p. 5
• WER measures errors over the full reference transcript, while Bias-WER measures errors on annotated contextual entities such as names, locations, and products.Source: §6.2 · p. 5
• PHI-4-MULTIMODAL serves as the backbone in all experiments.Source: §6.3 · p. 5
• Raw-context ASR training samples between zero and ten context turns independently for each example, exposing the model to variable-length contexts.Source: §6.3 · p. 5
• Abstract Compression initializes Stage 1 from the best single-turn ASR model, freezes that model, and optimizes only the attached compression module.Source: §6.3 · p. 5
• At inference, evaluation uses a fixed number of prior turns, supplies prior-turn transcripts as ground-truth text, and predicts only the current-turn transcript.Source: §6.3 · p. 6

Raw Context

• Multi-turn fine-tuning enables raw conversational context to improve recognition on in-domain and out-of-domain tests, especially for entities.Decoding with context without multi-turn training instead degrades performance.Source: §7.1 · p. 6

Abstract Compression

• 21.2% WER with K = 16 after LibriSpeech 960h plus DefinedAI training shows compressed tokens retain usable acoustic information, though full-resolution ASR remains better.With DefinedAI-only Stage 1 training, increasing K from 4 to 16 reduces DefinedAI WER from 51.2% to 37.1%.Source: §7.2 · p. 6, Table 2 · p. 7
• 13.3% Bias-WER on DefinedAI and 24.5% on WoW show compressed context improves over single-turn baselines but remains behind raw context.The corresponding single-turn and raw-context values are 13.5% and 13.1% on DefinedAI, and 25.6% and 23.3% on WoW.Source: §7.2 · p. 7
• Abstract Compression replaces each prior-turn audio sequence with K latent tokens while retaining prior-turn transcripts explicitly.This targets audio as the dominant source of contextual token cost.Source: §7.3 · p. 7

Compression Rate Analysis

• As context length grows, audio and overall compression rates decrease and stabilize because each added turn contributes exactly K latent audio tokens.The variance of both ratios also narrows with more turns, while the metrics exclude the current utterance’s full-resolution audio.Source: §7.3 · pp. 7–8, Figure 2 · p. 7

Ablation Studies

• K = 16 yields the strongest WoW contextual result, reducing Bias-WER from 26.5% without context to 24.5% with 5 or 10 turns.Most contextual gains arise within the first few turns and largely saturate by 5 turns; stronger Stage 1 single-turn alignment does not necessarily improve downstream contextual ASR.Source: §7.4 · p. 8, Table 3 · p. 8

• Abstract Compression replaces prior-turn audio with a fixed latent budget while retaining transcripts explicitly.Source: §8 · p. 8
• The compressed representation recovers part of the benefit of full raw context while reducing the prior-turn audio footprint.Source: §8 · p. 8
• The results suggest compact conversational-audio representations offer a practical quality-efficiency trade-off for context-aware ASR.Source: §8 · p. 8

• Abstract Compression is evaluated only for audio compression with a single multimodal LLM backbone.Source: Limitations · p. 9
• The efficiency analysis measures input token budget and prompt-length reduction, not inference latency, memory usage, or KV-cache behavior.Source: Limitations · p. 9

• DefinedAI contains scripted agent-customer conversations and supplies the train, development, and test data for the main ASR experiments.Its splits contain 17k/559/2k utterances, respectively.Source: A Dataset Details · p. 11
• WoW is an internal real call-center dataset used only for out-of-domain evaluation, with none of the models trained on it.Its test set contains 1,465 utterances and 3,434 entity words.Source: A Dataset Details · p. 11
• LibriSpeech 960h is used to pre-train the audio compression module for Stage 1 single-turn experiments, not for Stage 2 contextual fine-tuning or evaluation.Source: A Dataset Details · p. 11
• DefinedAI is the in-domain benchmark and WoW is the out-of-domain benchmark; contextual ASR models are trained on DefinedAI unless otherwise noted.Source: A Dataset Details · p. 11