PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis

Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, Yang Feng

arXiv:2505.02625v1cs.CLcs.AIcs.SDeess.AS

TL;DR

Existing speech interaction pipelines suffer from error accumulation, high latency, and limited paralinguistic modeling, while prior end-to-end systems trade responsiveness against naturalness. LLaMA-Omni 2 addresses this with SpeechLMs integrating an LLM, speech understanding, and autoregressive streaming speech synthesis, outperforming previous state-of-the-art models while achieving latency under 600ms.

  • Problem

    Conventional multi-stage speech pipelines accumulate errors, incur high latency, and struggle to capture paralinguistic information; prior streaming approaches also face naturalness limitations.

  • Method

    LLaMA-Omni 2 builds SpeechLMs on Qwen2.5 models, equipping an LLM with speech understanding and streaming speech generation through autoregressive text-to-speech and causal flow matching components.

  • Results

    LLaMA-Omni 2 outperforms previous state-of-the-art SpeechLMs on spoken question answering and speech instruction following, with latency under 600ms for real-time interaction.

  • Takeaways & Limitations

    The results support high-quality real-time speech interaction from a unified SpeechLM across spoken question answering and speech instruction following tasks.

  • Takeaways & Limitations

    The model cannot currently generate speech responses with different styles, such as emotion or speech rate, from input content or paralinguistic information.

Abstract

from arXiv · show

Real-time, intelligent, and natural speech interaction is an essential part of the next-generation human-computer interaction. Recent advancements have showcased the potential of building intelligent spoken chatbots based on large language models (LLMs). In this paper, we introduce LLaMA-Omni 2, a series of speech language models (SpeechLMs) ranging from 0.5B to 14B parameters, capable of achieving high-quality real-time speech interaction. LLaMA-Omni 2 is built upon the Qwen2.5 series models, integrating a speech encoder and an autoregressive streaming speech decoder. Despite being trained on only 200K multi-turn speech dialogue samples, LLaMA-Omni 2 demonstrates strong performance on several spoken question answering and speech instruction following benchmarks, surpassing previous state-of-the-art SpeechLMs like GLM-4-Voice, which was trained on millions of hours of speech data.

1 Introduction

LLaMA-Omni 2 addresses the latency, error accumulation, and paralinguistic limitations of cascaded spoken-chatbot pipelines with a modular SpeechLM. It combines real-time responsiveness with natural speech generation through autoregressive streaming synthesis.

  • Motivation: Cascaded ASR–LLM–TTS pipelines can accumulate errors, increase latency through sequential processing, and miss paralinguistic information.End-to-end SpeechLMs are presented as a unified alternative requiring only small-scale speech fine-tuning.
  • Motivation: Real-time responsiveness and naturalness are both essential characteristics of spoken chatbots.LLaMA-Omni uses non-autoregressive streaming for low latency, but its speech can be less natural and fluent.
  • Approach: LLaMA-Omni 2 is a 0.5B–14B modular SpeechLM series built on Qwen2.5 models with Whisper’s encoder and an autoregressive streaming speech decoder.The design targets real-time, high-quality speech interaction while addressing limitations of prior streaming approaches.
  • Approach: A Read-R-Write-W strategy supports synchronized text and speech generation, while autoregressive decoding targets more natural speech than non-autoregressive synthesis.The strategy is introduced to address the trade-off between streaming latency and speech naturalness.

2 Model: LLaMA-Omni 2

LLaMA-Omni 2 equips a Qwen2.5-based language model with speech understanding and streaming speech generation modules. Its two-stage training and autoregressive speech pipeline jointly support incremental, low-latency response generation.

  • Architecture: Qwen2.5 serves as the core language model, while Whisper-large-v3 provides speech representations through a speech encoder and adapter.The adapter downsamples consecutive frames and maps them for language-model processing.
  • Streaming Speech Generation: A supervised speech tokenizer converts responses into discrete tokens, which an autoregressive text-to-speech language model generates from LLM outputs.MTTS uses a decoder-only Transformer with an expanded vocabulary containing speech tokens.
  • Streaming Speech Generation: The gate fusion module combines LLM hidden states with text embeddings before MTTS generates speech tokens.Hidden states provide contextual information, while text tokens provide precise textual content and alignment.
  • Streaming Speech Generation: Read-R-Write-W generates W speech tokens after every R fused representations, then continues generation after all representations are read.Training applies cross-entropy loss only to the generated speech tokens.
  • Streaming Speech Generation: A causal flow-matching model converts generated speech tokens into mel spectrogram chunks, and HiFi-GAN produces the final waveform.Every W generated speech tokens form one synthesis chunk.
  • Training and Inference: Training uses 200K multi-turn speech-to-speech dialogues in two stages, separately training speech-to-text and text-to-speech components before speech-to-speech training.Stage II freezes the speech encoder, adapter, and LLM while training the gate fusion module and MTTS.
  • Training and Inference: During inference, LLM text and hidden states feed incremental speech-token generation, flow matching, and vocoding so text and speech responses are produced simultaneously.The first speech chunk’s latency is modeled from the LLM, MTTS, flow-matching, and vocoder decoding times.

3 Data Construction

The authors construct 200K multi-turn text dialogues and convert them into speech-to-speech data with controlled voice diversity. Instructions vary voices across dialogues while responses maintain a consistent voice.

  • Text Dialogue Construction: The dataset extends InstructS2S-200K by generating 200K multi-turn dialogues from Alpaca and UltraChat using Llama-3.3-70B-Instruct.Turn counts are sampled from Poisson(λ = 2) and clipped to 1–5 turns.
  • Speech Synthesis: Instruction speech uses randomly generated voices, while CosyVoice2 voice cloning maintains consistency across turns within each dialogue.This combination provides voice diversity across dialogues and consistent instruction voices within dialogues.
  • Speech Synthesis: All response speech is synthesized with a uniform voice prompt using CosyVoice2-0.5B.Responses therefore share a consistent voice design across the constructed data.

4 Experiments

The experiments evaluate LLaMA-Omni 2 across spoken question answering and speech instruction following, measuring both speech-to-text and speech-to-speech capabilities. They also specify the model configurations, training procedure, evaluation metrics, and comparison baselines.

  • Model Configuration: LLaMA-Omni 2 uses Whisper-large-v3 encoding, Qwen2.5 Instruct models from 0.5B to 14B, a Qwen2.5-0.5B text-to-speech model, and CosyVoice 2 speech generation components.The speech adapter applies 5× downsampling and an FFN with intermediate dimension 2048; the read-write strategy is R = 3 and W = 10.
  • Training Details: Training uses 200K multi-turn speech-to-speech dialogues in a two-stage procedure with selectively frozen speech encoder, adaptor, LLM, and text-to-speech components.The supplied training description specifies Stage I(a), Stage I(b), and Stage II, including separate batch sizes, epochs, and peak learning rates.
  • Evaluation Tasks: The evaluation covers spoken question answering and speech instruction following, with both speech-to-text and speech-to-speech assessment.Speech-to-speech responses are transcribed with Whisper-large-v3 before applying the speech-to-text evaluation method.
  • Evaluation Tasks: Spoken question answering uses Llama Questions and Web Questions, with text-form Web Questions synthesized into speech using CosyVoice2-0.5B.
  • Evaluation Tasks: Speech instruction following evaluates 199 synthesized-speech instructions from the helpful_base and vicuna Alpaca-Eval subsets, excluding math and code instructions.
  • Evaluation Metrics: The instruction-following metrics include GPT-4o scoring, ASR-WER for text–speech consistency, UTMOS for speech naturalness, and first-speech-chunk latency.GPT-4o assigns scores from 1 to 5, while latency is measured on a single NVIDIA L40 GPU.

5 Results and Analysis

LLaMA-Omni 2 performs strongly across spoken question answering and speech instruction following while maintaining speech quality and real-time responsiveness. Ablations show that adaptive fusion, streaming TTS pretraining, read/write choices, and multi-turn data size materially affect performance.

  • Spoken Question Answering: LLaMA-Omni2-7B outperforms GLM-4-Voice and LLaMA-Omni in SpokenQA across both S2T and S2S settings.On Web Questions, its S2T-to-S2S drop is 3.2 (34.5→31.3), compared with 16.3 for GLM-4-Voice and 9.7 for LLaMA-Omni.
  • Speech Instruction Following: LLaMA-Omni2-3B/7B/14B outperforms GLM-4-Voice and LLaMA-Omni in speech instruction following under both S2T and S2S settings.Performance improves as LLM size increases, with LLaMA-Omni2-14B achieving significantly better performance.
  • Gate Fusion Module: The gate fusion ablation shows performance declines when adaptive fusion is removed and when text embeddings are removed entirely.The module adaptively fuses LLM hidden states with text embeddings before the text-to-speech language model.
  • TTS Pretraining: Streaming TTS pretraining outperforms offline TTS pretraining and direct text initialization, demonstrating the importance of TTS pretraining.The offline TTS strategy shows a slight performance drop, while direct text initialization shows a significant performance drop.
  • Read/Write Strategy: R = 3 and W = 10 provides a trade-off across ASR-WER, speech quality, and latency, with latency below 500ms without engineering optimizations.ASR-WER is lowest at R = 3 and W = 10; larger W improves UTMOS, while latency depends jointly on R and W.
  • Effects of the Training Data Sizes: Performance improves with larger training sets and stabilizes at 200K samples, while multi-turn dialogue data consistently outperforms single-turn data at equal sample counts.The results indicate that 200K multi-turn dialogue samples are generally sufficient while maintaining efficient training.

6 Related Work

SpeechLMs are broadly divided into native and modular designs, with recent work extending LLMs toward speech understanding, generation, and full-duplex interaction. LLaMA-Omni 2 is most closely related to modular systems using speech decoders and autoregressive streaming generation.

  • SpeechLM Taxonomy: SpeechLMs generally fall into native models that directly process speech tokens and modular models that add speech modules to pretrained LLMs.Native systems extend pretrained vocabularies with speech tokens, whereas modular systems attach speech-related components to LLMs.
  • Native SpeechLMs: Native SpeechLMs have explored speech-text interleaving, cross-modal transfer, and full-duplex conversation.Examples include Spirit-LM and GLM-4-Voice for interleaved training, and Moshi, OmniFlatten, and LSLM for full-duplex interaction.
  • Modular SpeechLMs: Modular SpeechLMs evolved from speech understanding toward speech generation through added speech decoders or simultaneous speech-token generation.LLaMA-Omni, Freeze-Omni, and OpenOmni add speech decoders, while Mini-Omni and SLAM-Omni generate speech tokens alongside text.
  • Closest Related Work: Minmo is the closest related work because it also uses an autoregressive streaming speech decoder, but it is trained on 1.4M hours of data versus LLaMA-Omni 2’s few thousand hours.The comparison positions LLaMA-Omni 2 as a more data-efficient training solution.

7 Conclusion

LLaMA-Omni 2 is a 0.5B–14B SpeechLM series combining an autoregressive text-to-speech language model with a causal flow matching model for real-time speech interaction. It outperforms previous state-of-the-art SpeechLMs on evaluated tasks and achieves latency under 600ms.

  • Conclusion: LLaMA-Omni 2 spans 0.5B to 14B parameters and targets real-time, high-quality speech interaction.Its streaming speech generation integrates an autoregressive text-to-speech language model with a causal flow matching model.
  • Conclusion: Latency under 600ms meets the paper’s stated requirements for real-time interaction.The paper also reports ablation studies examining factors affecting overall performance.

Limitations

LLaMA-Omni 2 currently cannot generate speech responses with different styles from input content or paralinguistic information because training used conventional speech-to-speech dialogue data.

  • Scope Boundary: The model cannot currently vary response emotion or speech rate according to input content or paralinguistic information.The limitation concerns speech-style control rather than the model’s general speech-generation capability.
  • Scope Boundary: This limitation follows from training exclusively on conventional speech-to-speech dialogue data.The authors identify the training distribution as the relevant scope constraint.
  • Future Direction: The authors plan to address style-controlled generation through further training with suitable data.They describe this as a data-driven direction enabled by the model’s end-to-end training.

Ethical Considerations

Because LLaMA-Omni 2 is built on LLMs, its outputs may contain factual errors or hallucinations. Practical deployment therefore requires checking outputs against applicable standards.

  • Ethical Risks: LLaMA-Omni 2 inherits LLM-associated risks, including factual errors and hallucination issues.The ethical concern applies to generated outputs because the system is built on LLMs.
  • Ethical Risks: The paper does not treat strong speech interaction performance as evidence that outputs are factually reliable.The stated risk remains applicable despite the model’s LLM foundation and reported capabilities.
  • Practical Safeguard: The authors recommend checking outputs during practical use for compliance with required standards.This recommendation is presented as an operational safeguard for deployment.

A Prompt

The prompt asks a model to evaluate speech-interaction responses from transcribed user instructions and model outputs, then return a JSON score without explanations.

  • The evaluation concerns models that receive speech input and produce speech output.
  • Both the user’s instruction and the model’s response are transcribed into text using ASR for scoring.
  • The evaluator rates each response based on the provided instruction and response transcriptions.
  • The required output is a JSON object containing a score, with no explanation.

B Detailed Latency

The detailed-latency section reports stage-specific response delays for LLaMA-Omni2 models, distinguishing language-model, speech-language-model, and flow-matching-plus-vocoder stages.

  • LLM latency measures the time required to generate the first R text tokens.
  • TTS latency measures the time required to generate the first W speech tokens.
  • FM+Voc latency measures the time required to generate the first speech chunk using the flow matching model and vocoder.
  • Table 6 presents detailed latency measurements for the LLaMA-Omni2 model series.
Loading 2505.02625v1…