Source-linked AI summary
Language Model Can Listen While Speaking
Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen
TL;DR
Speech language models remain turn-based and cannot handle real-time sound or interruptions. The paper introduces LSLM, an end-to-end model that listens while speaking; among three fusion strategies, Middle Fusion offers a good balance between speech generation and real-time interaction.
Problem
Existing speech language models are limited to turn-based conversations and cannot handle real-time sound or interruptions.
Source: §1 · p. 2Method
LSLM combines a token-based decoder-only TTS speaking channel with a streaming SSL listening channel and detects turn-taking in real time.
Source: §1 · p. 2Results
Middle Fusion achieves a good balance between speech generation and real-time interaction in the proposed LSLM.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
The paper frames interruption handling as a core capability for real-time interaction in interactive speech language models.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Dialogue serves as the most natural manner of human-computer interaction (HCI). Recent advancements in speech language models (SLM) have significantly enhanced speech-based conversational AI. However, these models are limited to turn-based conversation, lacking the ability to interact with humans in real-time spoken scenarios, for example, being interrupted when the generated content is not satisfactory. To address these limitations, we explore full duplex modeling (FDM) in interactive speech language models (iSLM), focusing on enhancing real-time interaction and, more explicitly, exploring the quintessential ability of interruption. We introduce a novel model design, namely listening-while-speaking language model (LSLM), an end-to-end system equipped with both listening and speaking channels. Our LSLM employs a token-based decoder-only TTS for speech generation and a streaming self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses both channels for autoregressive generation and detects turn-taking in real time. Three fusion strategies -- early fusion, middle fusion, and late fusion -- are explored, with middle fusion achieving an optimal balance between speech generation and real-time interaction. Two experimental settings, command-based FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity to diverse instructions. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems. This study aims to advance the development of interactive speech dialogue systems, enhancing their applicability in real-world contexts.
1 Introduction
Existing speech language models are largely limited to turn-based interaction, while text-centric duplex approaches rely on external speech modules. The paper introduces LSLM, an end-to-end model for listening and speaking simultaneously, and evaluates its fusion strategies and interruption behavior.
1 Introduction
- Prior speech models are limited to turn-based conversation, while text-centric duplex systems use external ASR and TTS modules and retain perceivable latency and limited paralinguistic ability.These constraints motivate treating full-duplex capability as intrinsic to speech language models.Source: §1 · p. 2
- LSLM combines listening and speaking channels in one end-to-end model to support real-time turn-taking while speaking.It uses a token-based decoder-only TTS and a streaming SSL encoder, and fuses both channels for autoregressive generation.Source: §1 · p. 2
- Command-based and voice-based tests indicate that LSLM handles noisy input and diverse interruption instructions with little impact on the previous system.The voice-based setting tests interruptions from unseen speakers using varied words.Source: §1 · p. 2
- The paper formulates Full Duplex Modeling for interactive speech language models as the task of simultaneous listening and speaking.LSLM is designed to listen to outside signals and provide feedback in real time while speaking.Source: §1 · p. 2
- Middle fusion provides the best reported balance among early, middle, and late fusion for speech generation and real-time interaction.The paper tests these three strategies for fusing duplex signals.Source: §1 · p. 2
2 Related Work
The related work distinguishes one-way simplex models, turn-based half-duplex models, and full-duplex systems that listen while speaking. It notes that existing full-duplex efforts and products leave open questions about speech language models.
2 Related Work
- The paper uses interactive speech language model for an SLM with full-duplex capability, distinguishing it from simplex and half-duplex systems.Figure 1 depicts the distinctions among these three model types.Source: §2 · p. 2, Figure 1 · p. 3
2.1 Simplex and Half Duplex Speech Language Model
- Simplex models communicate in one direction, whereas half-duplex models can listen and speak but perform only one action at a time.The paper associates half-duplex models with turn-based systems such as SpeechGPT, LauraGPT, and VioLA [48], [5], and [42].Source: §2.1 · p. 3
2.2 Full Duplex Speech Language Model
- Full-duplex speech language models can listen and speak simultaneously; existing efforts include text-centric systems with cascaded ASR and TTS, while open-source models and detailed analyses remain unavailable.The passage also identifies GPT-4o and Moshi as products with full-duplex spoken dialogue capability.Source: §2.2 · p. 3
3 Full Duplex Modeling (FDM)
The section contrasts turn-based autoregressive modeling with full-duplex modeling, where the next response token can use information from both channels. The listening sequence arrives in real time and unpredictably.
3 Full Duplex Modeling (FDM)
- In simplex or half-duplex autoregressive modeling, the next token depends on context and prior output within the current channel, without information from other channels.The training objective models response tokens autoregressively from context-response pairs.Source: §3 · pp. 3–4
- Full-duplex modeling predicts the next response token using both the speaking-channel history and information from the other channel.At inference, the current speaking output and newly acquired listening input are fed in simultaneously to influence the next speaking output.Source: §3 · p. 4
- The listening sequence in full-duplex modeling is produced in real time and unpredictably, so both channels can inform each next speaking step.The passage describes the speaking output and listening input at the preceding inference step as simultaneous inputs.Source: §3 · p. 4
4 Proposed LSLM
LSLM is designed to combine speaking and listening simultaneously, integrating speech generation, real-time audio input, and turn-taking detection in one model.
4.1 Speaking Ability
- A decoder-only token-based TTS generates speech from discrete audio tokens, while an acoustic prompt supplies the synthesized speech’s timbre.The paper notes that single-layer discrete tokens avoid waiting for non-autoregressive operations after autoregressive synthesis.Source: §4.1 · pp. 4–5
4.2 Listening Ability
- The listening channel encodes incoming audio into continuous features and projects them into a space the autoregressive model can process.Input audio may include environmental noise or human speech.Source: §4.2 · p. 5
4.3 FDM Ability
- The model conditions speech generation on prior speaking tokens and processed listening information, using streaming audio features during inference.The SSL encoder must be streaming to provide real-time audio features.Source: §4.3 · pp. 5–6
- An added IRQ token lets LSLM terminate early when turn-taking occurs, with the model targeting interruption detection within interval µ.The listening channel and speaking channel are integrated using early, middle, or late fusion.Source: Figure 2 · p. 5, §4.3 · p. 6
- Early fusion combines channels at input embeddings, middle fusion at each Transformer block, and late fusion at output logits before softmax.Figure 3 presents these three listening-channel integration designs.Source: §4.3 · p. 6, Figure 3 · p. 6
5 Setup
The setup specifies LSLM’s architecture, the speech and interruption data used for evaluation, and the training and inference procedure.
5.1 Model Details
- LSLM uses a 106M-parameter, 12-block decoder-only Transformer and a 34M-parameter streaming vq-wav2vec encoder for audio features.A linear projection adapts listening features to the autoregressive model, and a GAN-based vocoder recovers speech waveforms.Source: §5.1 · p. 6
5.2 Data Details
- Training and evaluation use LibriTTS speech, noise from MUSAN, a “Honey” command, and single-word Speech Commands with an 8:1:1 split.The voice-based test speakers are absent from training; LibriTTS-testsetB includes 500 utterances from 37 unseen speakers.Source: §5.2 · p. 7, Table 1 · p. 7
5.3 Training and Inference Details
- Training runs for 20 epochs, adds noise and interruption with 50% probability each, and selects the lowest-validation-loss checkpoint.Optimization uses AdamW with a maximum learning rate of 5 × 10−4, no weight decay, and batch size 4.Source: §5.3 · p. 7
6 Experiments
The experiments evaluate speech-generation quality and real-time turn-taking across command-based and voice-based settings, then assess training choices for middle fusion. Middle fusion performs best among the tested command-based architectures, while voice-based testing and ablations expose additional challenges and training effects.
6.1 Evaluation Metrics
- Interactive success is stopping speech within one second of interruption onset; Precision, Recall, and F1 summarize turn-taking outcomes, while WER evaluates generated speech against its text.The interactive evaluation distinguishes interruption hits and misses from false hits and true negatives.Source: §6.1 · pp. 7–8
6.2 Experiments results
- Evaluation uses 500 uninterrupted LibriTTS-testsetB utterances for TTS and 1,000 interactive utterances, split evenly between interruption and no-interruption cases.Listening-channel conditions are Clean or Noise; Vanilla TTS has an inherently clean input because it has no listening channel.Source: §6.2 · p. 8
- LSLMMF achieves 4.05% WER in clean conditions and 4.51% WER in noisy conditions; LSLMLF’s noisy-condition F1 falls to 94.89%.The authors conclude that middle fusion performs best overall among the tested command-based architectures, with competitive interactive performance.Source: §6.2 · p. 8, Table 2 · p. 9
- In voice-based FDM, LSLM has 5.33% WER in clean conditions and 8.50% WER in noisy conditions; noisy-condition Precision, Recall, and F1 are 87.69%, 82.77%, and 85.15%.This setting uses diverse interruption commands and unseen speakers, which the authors identify as adding complexity.Source: §6.2 · p. 8, Table 3 · p. 9
- Without interruption, IRQ-token probability stays below 1 × 10^-3; after a real-time turn-taking signal, it rises until the token is sampled.Figure 4 displays IRQ-token probability over time on a logarithmic scale.Source: §6.2 · p. 9, Figure 4 · p. 9
6.3 Ablation Study
- Continuing training both the TTS backbone and streaming SSL encoder yields 4.05% WER, 97.80% Precision, 98.19% Recall, and 98.00% F1.The authors report that the SSL encoder performs better when continued training than when its parameters are fixed; they suggest limited noise exposure during pre-training as one possible reason.Source: §6.3 · p. 10, Table 4 · p. 10
7 Conclusion
The conclusion presents LSLM as an end-to-end model for real-time turn-taking that combines speech generation with streaming audio input. Middle fusion offers a balance between speech generation and interaction, while the work is described as an initial exploration with further capabilities left to develop.
7 Conclusion
- LSLM combines token-based decoder-only TTS with a streaming SSL encoder to listen and speak simultaneously and handle real-time turn-taking.The model fuses the two channels using early, middle, or late fusion.Source: §7 · p. 10
- Middle fusion demonstrates a superior balance between speech generation and real-time interaction among the explored fusion strategies.Source: §7 · p. 10
- Command-based and voice-based evaluations show duplex communication with minimal impact on system performance, including responsiveness to diverse instructions from unseen speakers and noisy environments.Source: §7 · p. 10
- The paper describes this as an initial exploration; future work includes full-duplex speech-in/speech-out dialogue, speaker-following, and audiovisual co-guidance.Source: §7 · p. 10