Source-linked AI summary
Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models
Bella Godiva, Yeonju Kim, Yong Man Ro
TL;DR
Full-duplex speech dialogue systems can lose coherence when audio is corrupted, while prior visual approaches often require costly multimodal retraining. AV-STE instead restores semantic speech tokens from noisy audio and lip video before a frozen speech LLM. Integrated with frozen Moshi, it improves response coherence under speaker interference and transfers gains to out-of-domain noisy evaluation, with limitations under visual corruption and domain shift.
Problem
Full-duplex dialogue models rely primarily on audio and are vulnerable to acoustic interference, while adapting them to visual input requires costly multimodal training.
Method
AV-STE uses streaming audio-visual encoding and noise-adaptive fusion to restore corrupted semantic tokens before a pretrained speech LLM, which remains frozen.
Results
AV-STE improves semantic-token recovery across noisy and interfering-speaker conditions, with gains transferring to the out-of-domain Seamless Interaction dataset.
Takeaways & Limitations
Visual speech cues can improve robust full-duplex dialogue through token enhancement without retraining the underlying speech LLM.
Takeaways & Limitations
AV-STE assumes reliable visual input, remains untested under visual corruption or missing video, and shows sensitivity to domain shift.
Abstract
from arXiv · showhide
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.
1 Introduction
Full-duplex dialogue models are vulnerable to acoustic interference because they rely primarily on audio, while visual speech cues can complement corrupted representations. AV-STE addresses this gap by restoring semantic speech tokens before a frozen speech LLM, improving noisy dialogue robustness without retraining it.
- Background noise and competing speakers corrupt user-speech representations, degrading understanding and response coherence in full-duplex dialogue.
- Visual lip movements provide complementary information for ambiguous or corrupted speech, but conventional AVSR predicts text rather than speech tokens.Using AVSR would add speech-to-text and text-to-speech-token conversion stages.
- Existing audio-visual dialogue approaches require expensive, data-intensive multimodal training that adapts the pretrained dialogue model to a new modality.
- AV-STE restores corrupted semantic speech tokens from noisy audio and lip video before they reach the dialogue model.It uses a streaming audio-visual encoder and noise-adaptive fusion while leaving the underlying speech LLM frozen.
- AV-STE improves semantic-token recovery and noisy response coherence across same-dataset interference and out-of-domain conditions while preserving turn-taking behavior.
2 Related Work
Related work establishes full-duplex speech-token dialogue, explores visual input, and motivates AV-STE’s modular alternative. AV-STE enhances semantic tokens directly while retaining acoustic streams and avoiding retraining the speech LLM.
- Full-duplex speech LLMs process streaming user speech while generating responses, enabling interruption and backchanneling through more natural turn-taking.
- Moshi operates directly over speech-token streams within a unified full-duplex architecture, providing the setting that AV-STE improves.
- MoshiVis adds visual input for image understanding, whereas AV-STE uses continuous lip video to recover speech information under acoustic corruption.
- AV-Dialog trains the dialogue model on paired audio-visual conversations, while AV-STE restores corrupted semantic tokens before they reach a frozen model.
- AV-STE enhances the semantic-token stream while retaining acoustic-token streams unchanged, allowing direct consumption by Moshi.
3 Audio-Visual Speech Token Enhancer (AV-STE)
AV-STE is a streaming front-end that combines audio-visual features with uncertain speech-token predictions to produce enhanced semantic tokens. Its noise-adaptive fusion balances visual and tokenizer information while preserving low-latency inference and unchanged acoustic streams.
- 3.1 Overview: AV-STE enhances semantic speech tokens because they primarily encode linguistic content critical for speech understanding and coherent response generation.Lip movements provide phonetic cues but limited information about timbre, prosody, and background noise.
- 3.1 Overview: The framework predicts clean semantic tokens from noisy semantic tokens and synchronized facial video before feeding them to a frozen speech LLM.
- 3.1 Overview: A Streaming Audio-Visual Encoder extracts visual features, while Noise-Adaptive Fusion combines them with noisy tokenizer representations before direct delivery to the frozen dialogue model.
- 3.2 Streaming AV Encoder: Causal convolutions and bounded-lookahead attention support streaming inference; the main configuration uses L = 4, corresponding to 160 ms at 25 Hz.
- 3.3 Noise-Adaptive Fusion: Soft token cross-attention preserves tokenizer uncertainty instead of relying on potentially misleading hard argmax selections.
- 3.3 Noise-Adaptive Fusion: Visual features serve as queries and soft token embeddings as keys and values, allowing visual cues to retrieve information from corrupted speech-token representations.
- 3.3 Noise-Adaptive Fusion: The modulation gate uses tokenizer uncertainty to adaptively weight speech-token and cross-attended audio-visual representations under different noise conditions.Lower normalized entropy indicates higher estimated tokenizer reliability, while higher entropy indicates greater uncertainty.
4 Experimental Setup
Experiments use LRS3 for target speech and in-domain training, AudioSet for noise and interference, and Seamless Interaction only for out-of-domain evaluation. Evaluation uses single-turn utterances across five SNR levels, with AV-STE integrated into frozen Moshi and Mimi semantic tokens.
- Datasets and preprocessing: LRS3 provides the target-speech data for training and in-domain evaluation, with cropped 96×96 grayscale mouth frames aligned to 16 kHz audio.
- Datasets and preprocessing: AudioSet supplies environmental non-speech noise and conversational speech for cross-dataset speaker interference.
- Datasets and preprocessing: Seamless Interaction is reserved for out-of-domain evaluation without fine-tuning or training data use.
- Training and evaluation data synthesis: Initial training mixes clean speech, non-speech noise, and conversational speech in a 20/40/40 ratio with SNR uniformly sampled from −10 to 10 dB.
- Training and evaluation data synthesis: Same-dataset training replaces AudioSet interference with 1–4 randomly selected LRS3 speakers while retaining the same sampling ratio and SNR range.
- Evaluation: Evaluation selects 125 single-turn utterances each from LRS3 and Seamless Interaction, while Table 1 reports semantic-token accuracy across five SNR levels.
5 Experimental Results and Analysis
AV-STE improves semantic-token recovery across noise and interference conditions, with especially large gains under severe corruption and same-dataset multi-speaker interference. These token-level improvements transfer to more coherent full-duplex responses while largely preserving turn-taking and latency.
- Semantic-token recovery: 59.92% versus 4.81% semantic-token accuracy at −10 dB non-speech noise demonstrates the largest reported severe-corruption gain.The result supports complementary visual information when the acoustic representation is heavily corrupted.
- Same-dataset speaker interference: 65.60% versus 12.61% average accuracy under same-dataset multi-speaker interference shows robust recovery after further LRS3 interference training.At −10 dB, AV-STE maintains 56.18% accuracy while Mimi falls to 1.31%.
- Out-of-domain generalization: 36.76% versus 16.27% and 34.96% versus 15.80% noisy accuracy on Seamless Interaction show gains under out-of-domain non-speech noise and speaker interference.Clean accuracy decreases from 76.78% on LRS3 to 43.79% on Seamless, indicating sensitivity to domain shift.
- Ablation and modality: Audio-visual S-AVH outperforms audio-only S-AVH, while Soft-CA+NAM achieves the highest average accuracy among fusion variants.The final configuration uses Soft-CA+NAM.
- Ablation and modality: Audio-visual fusion provides increasing advantage as SNR decreases, and L=4 yields the best accuracy–latency trade-off.The selected lookahead is 4 frames, or 160 ms.
- Full-duplex dialogue robustness: 1.91 versus 1.42 average GPT coherence under same-dataset speaker interference shows improved responses with frozen Moshi.AV-STE performs best from 5 to −10 dB, while TOR and latency remain close to Moshi under noisy conditions.
- Full-duplex dialogue robustness: 2.10 versus 1.58 and 2.03 versus 1.50 average noisy coherence on Seamless Interaction show downstream gains under out-of-domain non-speech noise and speaker interference.No Seamless Interaction data is used for training.
6 Conclusion
AV-STE improves full-duplex spoken dialogue robustness by recovering corrupted semantic tokens before frozen Moshi, without retraining the underlying speech LLM. These gains extend to out-of-domain data and preserve turn-taking behavior.
- AV-STE improves full-duplex spoken dialogue robustness without retraining the underlying speech LLM.It recovers corrupted semantic tokens before they are consumed by frozen Moshi.
- AV-STE substantially improves semantic-token recovery across non-speech noise, speaker interference, and same-dataset multi-speaker interference.The largest gains occur under severe acoustic corruption.
- AV-STE improvements transfer to the out-of-domain Seamless Interaction dataset.
- Enhanced tokens improve spoken-response coherence across noisy conditions without degrading turn-taking behavior.
7 Limitations
AV-STE assumes reliable visual input and remains sensitive to domain shift and clean-speech representation mismatch.
- AV-STE assumes reliable visual input, so robustness to visual corruption, occlusion, and missing video remains unexplored.
- Lower clean-token accuracy on out-of-domain Seamless Interaction indicates sensitivity to domain shift.
- Predicted semantic tokens do not exactly reproduce native Mimi tokens, introducing a small representation mismatch on clean speech.
A Cross-Attention Direction Ablation
The ablation evaluates cross-attention direction and supports using audio-visual features as queries over corrupted speech-token representations.
- AV features serve as queries, while soft speech-token embeddings serve as keys and values.This arrangement lets noise-robust audio-visual cues guide retrieval from corrupted token representations.
- Reversing the cross-attention roles degrades WER under both noise conditions.The comparison is reported in Table 6, where lower WER is better.
B Detailed Semantic Token Accuracy Results
Detailed per-SNR results show that AV-STE improves semantic token accuracy over the audio-only Mimi baseline across noise conditions, with the largest gains at low SNRs.
- AV-STE improves semantic token accuracy over the audio-only Mimi baseline across noise conditions.Table 7 reports the full per-SNR results for each noise type.
- The largest semantic-token accuracy gains occur at low SNRs.
C Detailed WER Results
AV-STE is evaluated through WER across noise conditions and GPT-4o relevance scores for successful takeovers. The evaluation also defines how takeovers, latency, and response relevance are measured.
- Detailed WER Results: AV-STE improves WER over the audio-only Mimi baseline across noise conditions, with the largest gains at low SNRs.The detailed results are reported per SNR and noise type.
- Take Over Rate: Takeovers are identified from ASR-derived utterances using a duration threshold of 1.0 s or a word-count threshold of 3.The supplied definition also distinguishes takeovers from mere backchannels.
- Latency: Latency is measured as mean signed delay over successful takeovers, with negative values clipped to zero for overlaps up to 0.5 s.This makes the metric reflect delay after the user’s turn ends rather than rewarding early interruptions.
- GPT-based Coherence Evaluation: GPT-4o rates response relevance from 1 to 5 using the user utterance and generated response, and the final score is their mean.The judge evaluates relevance only, not fluency, politeness, grammar, verbosity, factuality, or timing unless they directly affect relevance.
E Qualitative Evaluation
The qualitative evaluation compares AV-STE with Moshi using representative response examples and GPT scores. The examples assess whether generated responses better match the user’s utterance under noisy conditions.
- Qualitative Examples: AV-STE produces responses that better match the user’s utterance than the Moshi baseline in representative noisy examples.Moshi often generates generic or off-topic responses under noise, whereas AV-STE responses are more aligned with the user’s utterance.
- Related Quantitative Evaluation: Semantic-token accuracy and WER are reported separately across noise types and SNR levels, with lower WER being better.Mimi clean accuracy is 100% by definition because clean tokens serve as the reference targets.
- Qualitative Examples: Table 9 reports GPT scores for qualitative examples comparing AV-STE with Moshi.The table caption identifies the scores as GPT scores.