Source-linked AI summary
Acoustic-to-Text KV Compression for Full-Duplex Speech Models
Yejin Lee, Seungbeom Kim, Yongha Lee, Kyuhong Shim
TL;DR
Full-duplex models accumulate memory-intensive acoustic history, and compression after input completion cannot reduce the peak already incurred during streaming. The paper uses listening-time slack for a transcription side channel, retaining text as older acoustic states are evicted. On ten-minute sessions, peak streaming KV usage falls by 64.6% versus the same adapted model without eviction, with improved long-form understanding and comparable interaction performance.
Problem
Full-duplex models accumulate memory-intensive acoustic history, and post-input compression cannot reduce the peak memory already incurred during streaming.
Source: §1 · p. 1Method
A listening-time transcription side channel preserves textual memory as older acoustic KV states are evicted, with LoRA transcription training and distillation to constrain native behavior.
Source: §1 · p. 1Results
Takeaways & Limitations
Short-utterance adaptation supports ten-minute speech processing without long-form training.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Full-duplex speech language models continuously accumulate acoustic key-value (KV) states, making long-running interactions memory-intensive. During listening, the model can finish processing an audio unit before the next arrives; we term the remaining interval listening-time slack. We propose acoustic-to-text KV compression, which introduces a transcription side channel to convert incoming speech into compact textual memory within this interval. When the cache exceeds a target budget during inference, older acoustic states are evicted while transcripts and recent acoustic context remain. We train the side channel with LoRA using cross-entropy on transcription segments. To preserve listening and speaking behavior, we apply knowledge distillation to the original model's token-level output distributions at native prediction positions. On ten-minute LongSpeech sessions, our MiniCPM-o 4.5 implementation reduces peak streaming KV-cache size by 64.6% compared with the same model without eviction. The proposed method also improves transcription, temporal question answering, and summarization over the baseline. Full-Duplex-Bench evaluations further show comparable pause-handling, turn-taking, and interruption performance.
1. INTRODUCTION
Acoustic histories make long-running full-duplex interaction memory-intensive, while compression after input completion cannot lower peak usage. The paper adds a transcript side channel during listening and evicts older acoustic states while retaining text and recent audio.
1. INTRODUCTION
- Listening-time slack provides about 900 ms for incremental ASR, and transcripts use approximately 3 sequence positions per second versus 17 for acoustic history.MiniCPM-o 4.5 processes one-second audio units.Source: §1 · p. 1
- The side channel records speech as textual memory during listening; when the cache exceeds its budget, older acoustic states are evicted while transcripts and recent acoustic context remain.Transcript tokens stay in context but are excluded from spoken output.Source: §1 · p. 1, Figure 1 · p. 2
- LoRA trains the side channel with transcription supervision, while token-level distillation from a frozen teacher constrains changes to native listening and speaking behavior.The adaptation uses short transcribed utterances, avoiding long-form speech training or full-duplex interaction data collection.Source: §1 · p. 1
- 64.6%: peak streaming KV-cache size decreases on ten-minute LongSpeech sessions versus the same adapted model without eviction, alongside real-time processing and comparable interaction behavior.Processing averaged 461 ms per one-second audio unit, with no deadline misses among 10,039 measured units.Source: §1 · p. 1
2. RELATED WORK
Related work spans full-duplex interaction models, speech and text KV compression, and methods that compute while listening. This paper distinguishes its approach by retaining textual memory for acoustic eviction during full-duplex streaming.
2.1. Full-Duplex Speech Models
- Full-duplex systems include synchronized speech chunks, parallel user and agent streams [3], and shared-temporal-axis alignment in MiniCPM-o 4.5 [8].Full-Duplex-Bench evaluates turn-taking, backchannel, and interruption behavior.Source: §2.1 · p. 2
2.2. KV Cache Compression
- Text-cache methods retain attention-sink and recent-window tokens, select states by attention, or compress during input; speech methods retain compressed speech representations [5] [6] [16], unlike this textual memory.The proposed method builds textual memory during listening and evicts older acoustic states during full-duplex streaming.Source: §2.2 · p. 2
2.3. Computation During Listening
- Prior listening-time computation targets reasoning or response quality [21]; this work uses lightweight LoRA fine-tuning to retain textual memory after acoustic KV eviction.The side channel is added to an existing full-duplex model.Source: §2.3 · p. 2
3. ACOUSTIC-TO-TEXT KV COMPRESSION
The section defines a transcription side channel as textual memory for speech and details its supervision and budget-triggered eviction procedure.
3.1. Transcription Side Channel
- Each duplex unit can append a delimited transcript during listening-time slack; the text stays in context but is excluded from spoken output.The transcript follows the native decision in both LISTEN and SPEAK units.Source: §3.1 · pp. 2–3
3.2. Transcription Supervision and Behavior Distillation
- Reference words are assigned to transcription targets by acoustic endpoint and an additional emission delay, with each word appearing in exactly one segment.The corresponding target unit is determined from the word's endpoint and delay.Source: §3.2 · p. 3
- A frozen teacher supplies distributions at native prediction events while transcription cross-entropy trains the side channel; rank-16 LoRA adapts the language model.Predictions align by native events rather than sequence indices because inserted transcription shifts student positions.Source: §3.2 · p. 3
- The four-term objective combines transcription loss with native-position distillation and group-balanced auxiliary losses for decisions and turn boundaries.The auxiliary losses average within groups and then across the groups present.Source: §3.2 · p. 3
3.3. Online KV Eviction
- When resident KV positions exceed budget B, eviction removes eligible states from units older than the five-unit retention window while retaining transcripts and response text.Position IDs advance with every fed token, so retained keys keep their original rotary positions without re-encoding.Source: §3.3 · pp. 3–4
4. EXPERIMENTAL RESULTS
The experiments evaluate long-form speech understanding, full-duplex interaction, and streaming memory and latency. They report reduced KV-cache use alongside speech-task and interaction results.
4.1. Training
- The model is fine-tuned on 132K short utterances from LibriSpeech train-clean, using LoRA rank 16 and α = 32 for three epochs.The 43.7M added parameters are 0.53% of the language model; word alignments come from MFA with an emission delay of δ = 0.5 s.Source: §4.1 · p. 4
4.2. Evaluation
- LongSpeech evaluation uses the first 100 English ten-minute sessions per task; Full-Duplex-Bench evaluation uses all 727 samples across five tasks.LongSpeech tasks are transcription (WER), temporal QA (accuracy), and summarization (BLEU and ROUGE); QA accuracy is judged by GPT-4o.Source: §4.2 · p. 4
4.3. Long-form Speech Understanding
- WER is 12.9 versus 96.9 and temporal QA accuracy is 42% versus 30% against native streaming, with improved summarization.The method achieves comparable QA and summarization performance to external-ASR cascades, while FastConformer and Whisper-large-v3 achieve lower WER with more added parameters.Source: §4.3 · p. 4
4.4. Full-Duplex Interaction
- Table 2 reports comparable pause-handling, turn-taking, and interruption, with backchannel takeover 0.218 versus 0.000; frozen-teacher supervision works without full-duplex interaction data.Without distillation, the model takes over nearly all pauses and backchannels and starts speaking from the first unit in almost every sample.Source: §4.4 · p. 4
4.5. Streaming Memory and Latency
- At a 4K budget, peak streaming KV size falls from 11.35K to 4.02K positions (64.6%) versus the same adapted model without eviction; floor eviction raises WER to 19.7.The authors identify 4K as the operating point.Source: §4.5 · p. 4
- At 4K, processing averages 461 ms per unit (99th percentile 675 ms) with no deadline misses across 10,039 units; maximum word-emission delay of 2.59 s fits the five-second retention window.Mean word-emission delay is 1.01 s versus 9.5 s for native prompted transcription.Source: §4.5 · p. 4
5. CONCLUSION
The conclusion presents acoustic-to-text KV compression as a way to reduce streaming memory while retaining long-form understanding and full-duplex interaction performance.
5. CONCLUSION
- The method reduces peak streaming KV usage by 64.6% on MiniCPM-o 4.5 versus the same adapted model without eviction, alongside improved long-form understanding over native streaming.The conclusion also reports real-time processing and comparable pause-handling, turn-taking, and interruption performance.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study used existing publicly available datasets and models and collected no new data.
6. COMPLIANCE WITH ETHICAL STANDARDS
- No new data were collected, and no ethical approval was required.The study used existing publicly available datasets and models.Source: §6 · p. 5