Source-linked AI summary
Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR
Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke
TL;DR
Standard LLM-based ASR often processes utterances in isolation, limiting its use of conversational context, while raw prior-turn audio creates a rapidly growing context cost. The paper introduces Abstract Compression, which replaces prior-turn audio with fixed learned latent tokens while retaining transcripts, and finds that supervised multi-turn training enables compressed context to recover part of raw-context gains across in-domain and out-of-domain evaluation.
Problem
Standard ASR commonly processes utterances independently, limiting conversational context use where contextual entities are especially error-prone.
Method
Abstract Compression replaces each prior-turn audio sequence with a fixed set of learned latent tokens while retaining prior-turn transcripts explicitly.
Results
After supervised multi-turn training, compressed context recovers part of raw-context conditioning’s gains across in-domain and out-of-domain evaluation, with benefits clearest for contextual entities.
Takeaways & Limitations
Compact conversational-audio representations offer a practical quality-efficiency trade-off for context-aware ASR.
Takeaways & Limitations
The evaluation covers only audio compression with a single multimodal LLM backbone and measures token-budget reduction rather than latency, memory, or KV-cache behavior.
Abstract
from arXiv · showhide
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
1 Introduction
Conversational context can help LLM-based ASR, especially for contextual entities, but raw prior-turn audio is costly. Abstract Compression retains transcripts explicitly while replacing prior-turn audio with fixed latent tokens.
- Conversational context matters because prior turns can introduce entities, pronunciations, and discourse cues needed to interpret utterances.
- Most ASR systems process utterances independently, often failing on rare contextual entities such as names, locations, and domain-specific terminology.
- After supervised multi-turn training, conversational context improves LLM-based ASR, with the clearest gains on contextual entities measured by Bias-WER.
- Abstract Compression replaces each prior-turn audio sequence with a fixed number of learned latent tokens while retaining the transcript explicitly.
- Across in-domain and out-of-domain evaluation, the compressed representation recovers part of raw-context conditioning’s gains with a smaller prior-turn audio footprint.
- The work contributes multimodal context analysis, Abstract Compression with two-stage training, and ablations of compression design choices.
2 Related Work
Prior contextual ASR work primarily injects text-side biasing signals, while multimodal LLM-based ASR has focused mostly on single utterances. This work connects conversation-aware recognition with efficient long-context compression.
- Contextual ASR has used contact names, locations, and domain-specific phrases to improve recognition of rare words and named entities.
- Recent multimodal ASR models combine pretrained LLMs with speech encoders or unified audio-text processing architectures.
- Most prior LLM-based ASR work emphasizes single-utterance recognition, instruction following, or general multimodal capability rather than multi-turn conversational context.
- Efficient sequence-modeling research reduces long-context cost through pruning, pooling, summarization, memory tokens, and latent bottlenecks.
- The paper studies richer multimodal conversation context for ASR and a task-driven bottleneck at the intersection of contextual ASR, multimodal LLMs, and efficient long-context modeling.
3 Multimodal LLM-Based ASR
The ASR system uses a multimodal LLM backbone that maps audio into the LLM’s input space and interleaves audio tokens with text. Its single-turn formulation provides the context-independent baseline.
- The experiments use PHI-4-MULTIMODAL’s speech-processing capabilities to isolate conversational context’s impact on speech recognition.
- An audio encoder and projection module map waveform inputs into the LLM’s input embedding space.
- Audio tokens are interleaved with text embeddings and processed by the PHI-4-MINI LLM.
- In the single-turn baseline, the model receives the current audio with a standard chat-template transcription instruction.
4 Context-Aware ASR
The paper tests whether prior conversational turns help ASR and finds that inference-time prompting alone is insufficient, whereas supervised multi-turn training improves contextual-entity recognition. Raw audio context increases prompt length rapidly.
- A conversation is modeled as audio segments and transcripts across N turns, with the final turn transcribed using the preceding N−1 turns as context.
- The inference-time setup prepends completed prior turns to the current transcription request using the model’s multi-turn chat format.
- Inference-time raw-context prompting degraded performance relative to the single-turn baseline, indicating that prompting alone was insufficient for reliable cross-turn use.
- Supervised multi-turn fine-tuning trains the model to predict the final transcript from prior turns and current audio.
- Raw-context fine-tuning improved Bias-WER and recovery of contextual entities, but high-resolution prior-turn audio makes total prompt length grow rapidly with conversation length.
5 Abstract Compression for Context-Aware ASR
Abstract Compression replaces each prior turn’s long audio-token sequence with fixed-length learned latent tokens while retaining its transcript explicitly. A two-stage training strategy aligns these compressed representations with the LLM and progressively teaches contextual transcription.
- Raw-context audio becomes increasingly costly with conversation length because each prior turn contributes many audio tokens, whereas transcripts remain comparatively compact.
- Abstract Compression maps each variable-length prior-turn audio sequence to K learned latent tokens and leaves the corresponding transcript explicit.
- Turn-specific learnable queries cross-attend to each turn’s audio tokens, producing latent vectors that capture that turn’s context through a fixed-length information bottleneck.
- Compression applies only to prior turns; the current utterance remains uncompressed to preserve its full acoustic detail.
- Stage 1 freezes the base model and trains the compression module for compressed-audio-to-LLM alignment, while Stage 2 jointly fine-tunes compression and audio LoRA parameters on compressed multi-turn inputs.
- Stage 2 uses a context-length curriculum that starts without prior turns and progressively increases the maximum number of available turns.
6 Experimental Setup
Experiments evaluate contextual ASR with standard WER and entity-focused Bias-WER on in-domain DefinedAI and out-of-domain WoW data. Training varies context length for raw-context models, while compressed-context models use a staged setup and curriculum.
- DefinedAI is the main in-domain dataset, WoW is the out-of-domain evaluation set, and LibriSpeech 960h is used only for Stage 1 compression training.
- WER measures errors over the full reference transcript, while Bias-WER measures errors on annotated contextual entities such as names, locations, and products.
- PHI-4-MULTIMODAL serves as the backbone in all experiments.
- Raw-context ASR training samples between zero and ten context turns independently for each example, exposing the model to variable-length contexts.
- Abstract Compression initializes Stage 1 from the best single-turn ASR model, freezes that model, and optimizes only the attached compression module.
- At inference, evaluation uses a fixed number of prior turns, supplies prior-turn transcripts as ground-truth text, and predicts only the current-turn transcript.
7 Experimental Results
After multi-turn fine-tuning, conversational context improves ASR, especially for contextual entities. Abstract Compression retains part of these gains with fixed-size prior-turn audio representations, while ablations expose accuracy–efficiency trade-offs.
- Raw Context: Multi-turn fine-tuning enables raw conversational context to improve recognition on in-domain and out-of-domain tests, especially for entities.Decoding with context without multi-turn training instead degrades performance.
- Abstract Compression: 21.2% WER with K = 16 after LibriSpeech 960h plus DefinedAI training shows compressed tokens retain usable acoustic information, though full-resolution ASR remains better.With DefinedAI-only Stage 1 training, increasing K from 4 to 16 reduces DefinedAI WER from 51.2% to 37.1%.
- Abstract Compression: 13.3% Bias-WER on DefinedAI and 24.5% on WoW show compressed context improves over single-turn baselines but remains behind raw context.The corresponding single-turn and raw-context values are 13.5% and 13.1% on DefinedAI, and 25.6% and 23.3% on WoW.
- Abstract Compression: Abstract Compression replaces each prior-turn audio sequence with K latent tokens while retaining prior-turn transcripts explicitly.This targets audio as the dominant source of contextual token cost.
- Compression Rate Analysis: As context length grows, audio and overall compression rates decrease and stabilize because each added turn contributes exactly K latent audio tokens.The variance of both ratios also narrows with more turns, while the metrics exclude the current utterance’s full-resolution audio.
- Ablation Studies: K = 16 yields the strongest WoW contextual result, reducing Bias-WER from 26.5% without context to 24.5% with 5 or 10 turns.Most contextual gains arise within the first few turns and largely saturate by 5 turns; stronger Stage 1 single-turn alignment does not necessarily improve downstream contextual ASR.
8 Conclusion
The paper finds that conversational multimodal context improves LLM-based ASR after supervised multi-turn training, especially for contextual entities. Abstract Compression reduces prior-turn audio to a fixed latent budget while recovering part of raw-context benefits, supporting a quality-efficiency trade-off.
- Abstract Compression replaces prior-turn audio with a fixed latent budget while retaining transcripts explicitly.
- The compressed representation recovers part of the benefit of full raw context while reducing the prior-turn audio footprint.
- The results suggest compact conversational-audio representations offer a practical quality-efficiency trade-off for context-aware ASR.
Limitations
The evaluation is limited to audio compression with one multimodal LLM backbone, and efficiency is assessed through input-token and prompt-length reduction rather than direct systems measurements.
- Abstract Compression is evaluated only for audio compression with a single multimodal LLM backbone.
- The efficiency analysis measures input token budget and prompt-length reduction, not inference latency, memory usage, or KV-cache behavior.
A Dataset Details
The paper evaluates in-domain DefinedAI and out-of-domain WoW data, and uses LibriSpeech 960h only to pre-train the Stage 1 audio compression module.
- DefinedAI contains scripted agent-customer conversations and supplies the train, development, and test data for the main ASR experiments.Its splits contain 17k/559/2k utterances, respectively.
- WoW is an internal real call-center dataset used only for out-of-domain evaluation, with none of the models trained on it.Its test set contains 1,465 utterances and 3,434 entity words.
- LibriSpeech 960h is used to pre-train the audio compression module for Stage 1 single-turn experiments, not for Stage 2 contextual fine-tuning or evaluation.
- DefinedAI is the in-domain benchmark and WoW is the out-of-domain benchmark; contextual ASR models are trained on DefinedAI unless otherwise noted.