PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA-Omni: Seamless Speech Interaction with Large Language Models

Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, Yang Feng

arXiv:2409.06666v2cs.CLcs.AIcs.SDeess.AS

TL;DR

Speech interaction with open-source LLMs remains insufficiently explored, especially for low-latency and high-quality responses. LLaMA-Omni combines speech processing, an LLM, and streaming speech decoding to generate text and speech directly and simultaneously. It achieves strong response quality and a minimum response latency of 236ms in the reported experiments.

  • Problem

    Open-source LLMs lack sufficiently explored methods for low-latency, high-quality speech interaction.

  • Method

    LLaMA-Omni combines a pretrained speech encoder, trainable speech adaptor, LLM, and streaming speech decoder to generate text and speech directly from speech instructions.

  • Results

    236ms minimum response latency is achieved in the streaming scenario, while LLaMA-Omni obtains the highest ChatGPT score among evaluated models in both S2TIF and S2SIF.

  • Takeaways & Limitations

    LLaMA-Omni provides a reported open-source approach for simultaneous text-speech interaction with strong content, style, and speech-quality results.

Abstract

from arXiv · show

Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.

1 INTRODUCTION

Speech interaction with open-source LLMs remains underexplored, while cascaded and intermediate-text systems trade latency against response quality. LLaMA-Omni addresses this with direct speech processing and simultaneous text-speech generation.

  • Motivation: Open-source LLMs lack well-explored approaches for low-latency, high-quality speech interaction.This gap matters because text-only interaction is unsuitable for some scenarios, while speech interaction can improve user experience.
  • Motivation: Cascaded ASR and TTS systems increase latency by sequentially producing transcription, text response, and speech response.Their pipeline first transcribes speech, then generates text, and finally synthesizes speech.
  • Motivation: Direct speech-to-speech models can reduce latency by avoiding intermediate text, but complex speech mapping often motivates intermediate-text generation.Intermediate text can improve generation quality while sacrificing some response latency.
  • Proposed approach: LLaMA-Omni combines a speech encoder, speech adaptor, LLM, and streaming speech decoder for speech interaction.The speech encoder and adaptor feed speech representations into the LLM, while the decoder generates speech from the LLM’s hidden states.
  • Proposed approach: During inference, LLaMA-Omni decodes text and corresponding speech units simultaneously from speech instructions.The architecture avoids first transcribing the speech into text and uses a non-autoregressive streaming Transformer speech decoder.

2 MODEL: LLAMA-OMNI

LLaMA-Omni combines speech encoding, adaptation, language modeling, and streaming speech decoding to generate text and speech directly from spoken instructions. Its two-stage training and non-autoregressive decoder support simultaneous low-latency output.

  • Architecture: LLaMA-Omni combines a speech encoder, trainable speech adaptor, LLM, and streaming speech decoder.The architecture maps spoken instructions into LLM embeddings and uses decoder outputs for speech generation.
  • Speech Encoder and Adaptor: Whisper encodes the speech instruction, while the adaptor downsamples consecutive frames and projects them through a two-layer perceptron.The speech encoder remains frozen during training.
  • Speech Decoder: The speech decoder upsamples LLM hidden states and uses CTC to align them with discrete speech units before vocoding the selected alignment.Inference selects the highest-probability alignment, collapses repeats and blanks, and synthesizes the waveform.
  • Training Strategy: Training uses two stages: speech adaptor and LLM training for text generation, followed by speech decoder training while the earlier components remain frozen.The first stage excludes the speech decoder; the second stage optimizes the decoder with the CTC objective.
  • Streaming Inference: Text prefixes produce corresponding speech alignments in parallel within chunks, so simultaneous text-and-speech decoding is not significantly slower than text-only generation.Waveform synthesis begins when the generated unit count reaches the predefined chunk size.

3 CONSTRUCTION OF SPEECH INSTRUCTION DATA: INSTRUCTS2S-200K

InstructS2S-200K converts existing text instruction data into speech-interaction training triplets through instruction rewriting, response generation, and speech synthesis. The resulting dataset contains 200K speech instructions drawn from Alpaca and UltraChat.

  • Dataset Pipeline: The dataset construction pipeline creates speech instruction, text response, and speech response triplets from existing text instruction data.The process addresses the limited availability of publicly available speech instruction data.
  • Instruction Rewriting: Instruction rewriting adds filler words, converts symbols into spoken forms, and makes instructions relatively brief for speech interaction.These changes are designed to reflect natural speech patterns and support TTS synthesis.
  • Response Generation: Response generation adapts text-based answers into concise yet informative speech responses rather than retaining lengthy text-oriented formats.The paper notes that speech responses typically avoid complex structures and non-verbal elements such as ordered lists or parentheses.
  • Speech Synthesis: Speech synthesis uses CosyVoice-300M-SFT for instructions and VITS trained on LJSpeech for responses.Instruction voices are randomly selected as male or female to make synthesized inputs more natural.
  • Dataset Composition: 200K examples combine around 50K Alpaca instructions with around 150K first-round UltraChat instructions in InstructS2S-200K.Alpaca covers diverse topics, while the selected UltraChat entries primarily contain questions about the world.

4 EXPERIMENTS

The experiments evaluate LLaMA-Omni on speech-to-text and speech-to-speech instruction following, including offline and streaming settings, using automated quality, alignment, naturalness, rate, and latency measures. LLaMA-Omni achieves the strongest reported response scores while supporting low-latency streaming, with trade-offs involving speech-text alignment and synthesis quality.

  • Evaluation setup: LLaMA-Omni is evaluated on S2TIF and S2SIF, with S2SIF tested in offline and streaming scenarios.The streaming setting generates speech simultaneously with text, whereas the offline setting synthesizes speech after generating the text response.
  • Evaluation setup: The evaluation uses ChatGPT Score, ASR-WER, UTMOS, WPS, and response latency to measure response quality, speech-text alignment, naturalness, speaking rate, and responsiveness.Latency is measured from speech-input arrival to speech-response onset on one NVIDIA L40 GPU.
  • Offline results: LLaMA-Omni’s ASR-WER is slightly higher than cascaded systems but remains acceptable, while SpeechGPT shows substantially poorer speech-text alignment.The authors relate LLaMA-Omni’s remaining alignment gap to training the speech decoder on approximately 1K hours, far less than industrial TTS data.
  • Streaming results: 236ms is LLaMA-Omni’s minimum streaming latency with Ω=10, below GPT-4o’s reported 320ms average audio latency.Increasing Ω slightly lowers ChatGPT Score while improving ASR-WER; smaller chunks reduce UTMOS because more synthesis segments create discontinuities.
  • Streaming results: At 563ms with Ω=100, LLaMA-Omni’s streaming metrics are relatively close to its offline results.The model maintains relatively stable performance across latency conditions.
  • Streaming results: Streaming SpeechGPT exceeds 4500ms latency, while cascaded systems can fall below 300ms but incur quality and speech-rate degradation at low latency.The authors associate cascaded systems’ reduced speech rate with more frequent pauses between words; LLaMA-Omni preserves end-to-end unit generation before vocoder conversion.

5 RELATED WORK

Related speech-language models generally either add speech tokens to an LLM or prepend a speech encoder and fine-tune for speech understanding. Streaming generation is important for reducing latency in interactive speech applications.

  • Speech/audio language models: Native multimodal speech-text models add speech tokens to an LLM’s vocabulary and continue pretraining on speech and text data.This approach typically requires substantial data and computational resources.
  • Speech/audio language models: Another approach adds a speech encoder before the LLM and fine-tunes the model for speech understanding tasks.Examples include speech recognition and speech translation systems.
  • Simultaneous generation: Streaming generation begins producing output before the entire input is received, supporting synchronization in applications such as speech recognition and simultaneous interpretation.For LLMs, streaming speech synthesis can reduce latency between the model and users.

6 CONCLUSION

LLaMA-Omni combines speech understanding, an LLM, and streaming speech generation for low-latency, high-quality interaction. It uses a 200K speech instruction dataset and achieves strong response quality, low latency, and rapid training.

  • LLaMA-Omni combines Llama-3.1-8B-Instruct with a speech encoder and streaming speech decoder to generate text and speech responses simultaneously.The architecture is designed for low-latency, high-quality speech interaction.
  • The InstructionS2S-200K dataset contains 200K speech instructions paired with speech responses for speech-interaction alignment.
  • Compared with previous speech-language models, LLaMA-Omni delivers superior responses in both content and style.
  • 236ms response latency and training in less than 3 days on 4 GPUs support rapid development of speech interaction models.

A PROMPT

The prompts define formats and constraints for generating speech-compatible instructions and responses, and for scoring speech interaction outputs. They emphasize concise, synthesizable language and structured JSON outputs.

  • The prompt template frames LLaMA-Omni as a helpful language and speech assistant that answers questions from user speech.
  • The ChatGPT-scoring prompt evaluates transcribed user instructions and model responses in speech interaction scenarios and returns a score without explanation.
  • Instruction-rewriting prompts convert written instructions into brief, human-like speech while avoiding content that cannot be synthesized by TTS.
  • The prompts request JSON outputs for instructions and responses, including the fields question and response.
  • Response-generation prompts require concise answers that avoid content unsuitable for text-to-speech synthesis.

B CASE STUDY

The case study compares model responses to a present-wrapping instruction. LLaMA-Omni provides a more detailed and helpful answer while maintaining a concise style suited to speech interaction.

  • Qwen2-Audio produces lengthy responses containing line breaks and parentheses unsuitable for speech synthesis.
  • For “How do I wrap a present neatly?”, the case study compares responses from Qwen2-Audio, SALMONN, SpeechGPT, and LLaMA-Omni.
  • SpeechGPT uses a speech-appropriate style but provides less information, while SALMONN’s response is somewhat long.
  • LLaMA-Omni’s response is more detailed and helpful while retaining a concise style, outperforming previous models in this speech-interaction example.

C NUMERICAL RESULTS IN THE STREAMING SCENARIO

The paper reports numerical streaming results for LLaMA-Omni, SpeechGPT, SALMONN + Orca, and Qwen2-Audio + Orca. These results are organized across Tables 4–7.

  • Tables 4–7 present numerical results for all evaluated models in streaming scenarios.
  • Table 4 reports LLaMA-Omni’s numerical results in the streaming scenario.
  • Table 5 reports SpeechGPT’s numerical results in the streaming scenario.
  • Table 6 reports numerical results for SALMONN + Orca in the streaming scenario.
  • Table 7 reports numerical results for Qwen2-Audio + Orca in the streaming scenario.
Loading 2409.06666v2…