PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation

Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan, Zhihao Du, Shiliang Zhang

arXiv:2410.17799v2cs.CLcs.AIcs.SDeess.AS

TL;DR

Turn-based spoken dialogue lacks simultaneous communication, while full-duplex systems still face challenges in achieving natural, low-latency interaction. OmniFlatten uses staged post-training and flattened dialogue data to adapt a text LLM; the reported results show reasonable dialogue quality and improved full-duplex capabilities from modality alignment and half-duplex learning.

  • Problem

    Full-duplex spoken dialogue aims to support simultaneous interaction, but achieving natural, low-latency exchanges remains challenging, especially with interruptions, backchannels, and overlapping speech.

    Source: §1 · p. 1

  • Method

    OmniFlatten progressively adapts a text LLM through modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning using flattened data.

    Source: §1 · p. 2

  • Results

    The reported dialogues have reasonable quality, and both modality alignment and half-duplex learning improve full-duplex dialogue capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present OmniFlatten as a straightforward full-duplex modeling scheme that does not change the text LLM backbone or require computationally intensive pre-training.

    Source: §5 · p. 8

    The work trains on Qwen2-0.5B and relatively little dialogue data; the model lacks user and assistant backchannel handling, and response speed needs further optimization.

    Source: §6 · p. 9

Abstract

from arXiv · show

Full-duplex spoken dialogue systems significantly surpass traditional turn-based dialogue systems, as they allow simultaneous bidirectional communication, closely mirroring human-human interactions. However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech. In this paper, we introduce a novel End-to-End GPT-based model OmniFlatten for full-duplex conversation, capable of effectively modeling the complex behaviors inherent to natural conversations with low latency. To achieve full-duplex conversation capabilities, we propose a multi-stage post-training scheme that progressively adapts a text large language model (LLM) backbone into a speech-text dialogue LLM, capable of generating text and speech in real time, without modifying the architecture of the backbone LLM. The training process comprises three stages: modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning. In all training stages, we standardize the data using a flattening operation, which enables unifying the training methods and the GPT backbone across different modalities and tasks. Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full-duplex spoken dialogue systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site (https://omniflatten.github.io/).

1 Introduction

Turn-based spoken dialogue does not support simultaneous speech and can struggle with interruptions, backchannels, and overlap; full-duplex systems aim for more natural, low-latency interaction. OmniFlatten adapts a text LLM through staged training and flattened dialogue data, with reported gains from modality alignment and half-duplex learning.

1 Introduction

  • Full-duplex systems support simultaneous two-way communication, but achieving low latency and natural interaction remains challenging.Turn-based systems may fall short on interruptions, backchannels, and overlapping speech.Source: §1 · p. 1
  • Unlike SyncLM’s deduplication strategy, OmniFlatten explores explicit text-token prediction to improve semantic capabilities.Both approaches learn interleaved chunks of user and assistant speech units for real-time full-duplex conversation.Source: §1 · p. 2
  • OmniFlatten uses ASR/TTS modality alignment followed by half-duplex and full-duplex dialogue learning, flattening dialogue data across stages to unify training across modalities and tasks.The approach adapts a text LLM without modifying its architecture or relying on computationally intensive pre-training.Source: §1 · p. 2
  • The results report reasonable dialogue quality and improved full-duplex capabilities from both modality alignment and half-duplex learning.The evaluation also covers assistant and user turn-taking and runtime efficiency.Source: §1 · p. 2

2 Related Work

Prior end-to-end spoken dialogue models often support speech input and output but remain turn-based. Full-duplex approaches differ in how they manage simultaneous streams, turn-taking, and audio-token sequences.

2 Related Work

  • SpeechGPT, LauraGPT, Mini-Omni, LLaMA-Omni, and GLM-4-Voice support speech and text inputs and outputs but are predominantly turn-based.Qwen-audio2 and SALMONN accept voice input but output text, relying on external TTS for speech.Source: §2 · p. 2
  • VITA uses separate response and environmental-monitoring modules, whereas Moshi models user speech and system text and speech streams in parallel.The parallel-stream approach is not natively supported by GPT models and requires designs such as acoustic delay and inner monologue.Source: §2 · pp. 2–3
  • SyncLM uses time-chunking and deduplicates audio sequences, while OmniFlatten retains discretized audio tokens and generates text and audio concurrently.OmniFlatten also uses ASR/TTS modality alignment and progressive learning; the passage says SyncLM produces only speech tokens.Source: §2 · p. 3

3 Methodology

OmniFlatten discretizes dialogue speech, interleaves it with text, and flattens the streams for a GPT-based model. Progressive training moves from modality alignment and turn-based dialogue toward chunked full-duplex speech interaction.

3 Methodology

  • The model interleaves discretized user and assistant speech tokens with corresponding text tokens, then flattens them into one sequence for progressive training.This adapts a text-based LLM into an end-to-end full-duplex spoken dialogue model.Source: §3 · p. 3

3.1 Audio Tokenization And Detokenization

  • The speech tokenizer discretizes audio into semantic tokens using an encoder and a 4096-code VQ codebook; OT-CFM converts tokens to Mel spectrograms for HiFiGAN audio generation.The tokenizer is supervised through multilingual ASR; the detokenization model is adopted from CosyVoice.Source: §3.1 · pp. 3–4

3.2 Modality Alignment

  • Modality alignment fine-tunes a pretrained text LLM on paired speech-text ASR and TTS samples to support speech understanding and generation.The resulting multimodal model is used for subsequent dialogue learning.Source: §3.2 · p. 4

3.3 Dialogue Learning

  • Dialogue learning proceeds from half-duplex training with speech and text streams to full-duplex training with progressively fewer streams.Full-duplex training removes the user text stream first, then the assistant output text stream, reducing dependence on text.Source: §3.3 · p. 4

3.3.1 Half-duplex Dialogue Training

  • Half-duplex training first transcribes user speech, generates a textual assistant response, and then predicts speech for that response.It precedes full-duplex learning because aligned ASR/TTS data have no overlap, making turn-based dialogue more consistent with modality alignment.Source: §3.3.1 · p. 4

3.3.2 Full-duplex Dialogue Training

  • Three-stream full-duplex training chunks and flattens input speech, output text, and output speech for real-time streaming without strict token-level alignment.Text chunks are 2 tokens and speech chunks are 10 tokens; the assistant text stream is later removed for two-stream speech-to-speech training.Source: §3.3.2 · pp. 4–5

4 Experiments

The experiments use speech-text alignment data and simulated multi-turn voice chats to train and evaluate OmniFlatten. They assess modality alignment, dialogue quality, and turn-taking performance.

4.1 Data

  • Modality-alignment training draws on about 100K hours of ASR and TTS data, approximately 30% open-source and 70% proprietary.The open-source collection includes Mandarin and English data; only 15% of Wenetspeech training data with high-quality transcripts is used for ASR.Source: §4.1 · p. 5
  • The simulated voice-chat pipeline retains about 390K turn-based sessions, synthesizes speech, simulates response and interruption timing, adds user-channel noise, and produces 2,000 hours of multi-channel dialogue.The noise is set to 15–30 dB SNR; 1% each is reserved for validation and testing.Source: §4.1 · p. 5

4.2 Training and Inference Setup

  • Training uses Qwen2-0.5B, cross-entropy, and sequence lengths of 1,024 tokens for modality alignment and 8,192 for dialogue learning.Dialogue learning masks user-channel loss; training uses AdamW for five epochs, and inference alternates predicted assistant speech and text with fixed user speech.Source: §4.2 · pp. 5–6

4.3 Evaluations

  • After modality alignment, OmniFlatten demonstrates performance in both ASR and TTS tasks, supporting its transition from a text LLM to a speech-text model.ASR is evaluated against Whisper models and TTS systems; TTS recognition uses WER for English and CER for Chinese, without standard MOS evaluation.Source: §4.3 · pp. 6–7
  • Multi-stage training improves performance among three-stream models; speech-only output drops significantly, and speech scores remain below text scores for dual-output models.The comparison also reports that omitting speech-text alignment yields the lowest score and omitting half-duplex learning further diminishes chat capability.Source: §4.3 · p. 7, Table 3 · p. 8
  • OmniFlatten scores above Moshi on English chat, below 7B LLaMA-Omni on English but above it on Chinese, and below 9B GLM-Voice; adding speech modality reduces chat capability against Qwen2-0.5B-Instruct.The authors attribute LLaMA-Omni’s Chinese proficiency primarily to speech-text alignment and suggest possible test-set leakage in GLM-Voice training.Source: §4.3 · pp. 7–8
  • OmniFlatten responds faster than Moshi in assistant and user turn-taking, but neither model has a high user turn-taking success rate within 25 tokens.The evaluation uses predefined user speech and a 1.5-second threshold for judging whether the assistant starts or stops speaking promptly.Source: §4.3 · p. 8, Table 4 · p. 8

5 Conclusion

OmniFlatten combines synthesized full-duplex dialogue data with multi-stage progressive training for modality alignment and dialogue learning. The approach leaves the text LLM backbone architecture unchanged and avoids computationally intensive pre-training.

5 Conclusion

  • OmniFlatten uses synthesized full-duplex spoken-dialogue data and progressive training for modality alignment and dialogue learning.Source: §5 · p. 8
  • The modeling scheme does not change the backbone text LLM architecture or rely on computationally intensive pre-training.Source: §5 · p. 8
  • Empirical evaluations describe the approach as promising for end-to-end models that handle full-duplex interactions.Source: §5 · p. 8

6 Limitations

The work trains on Qwen2-0.5B and a relatively small dialogue dataset. The authors identify chat capability, response speed, alignment objectives, and backchannel handling as limitations or areas for improvement.

6 Limitations

  • Training uses the Qwen2-0.5B model and a relatively small-scale dialogue dataset.Source: §6 · p. 9
  • The authors identify chat capability as an area with considerable potential for improvement, particularly compared with scaled-up similar models.Source: §6 · p. 9
  • Response speed needs further optimization, especially in user-turn-taking scenarios.Source: §6 · p. 9
  • The model does not yet handle more complex interaction phenomena such as user or assistant backchannels, and early stopping conflicts with some speech-text alignment objectives.Source: §6 · p. 9

A Prompt LLM for Dialogue Quality Evaluation

The prompt asks an LLM to evaluate a dialogue context and voice-system response using four criteria, then return strengths, weaknesses, an overall evaluation, and a score in JSON.

A Prompt LLM for Dialogue Quality Evaluation

Loading 2410.17799v2…