PaperlayerView exact revision on arXiv

Source-linked AI summary

Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM

Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, Long Ma

arXiv:2411.00774v5cs.SDcs.AIcs.CLeess.AS

TL;DR

Freeze-Omni addresses the difficulty of adding speech interaction to LLMs without large spoken datasets or catastrophic forgetting. It connects streaming speech modules to a frozen textual LLM through staged training, retaining the backbone’s capabilities while supporting low-latency speech-to-speech dialogue. The paper reports strong spoken-question-answering performance relative to the backbone and introduces duplex interruption handling, while identifying broader speech and downstream-task support as future work.

  • Problem

    Existing speech-text multimodal LLMs often fine-tune their backbones, while collecting million-hour-scale spoken Q&A data is difficult and can cause catastrophic forgetting.

    Source: §1 · p. 2

  • Method

    Freeze-Omni connects streaming speech input and output modules to a frozen text-modality LLM using three-stage training and chunk-level state prediction.

    Source: §2.1 · p. 2, §2.2.2 · p. 4

  • Results

    Freeze-Omni preserves the backbone’s intelligence in spoken question answering while achieving low-latency speech-to-speech dialogue with the LLM completely frozen.

    Source: §1 · p. 2, §3.4 · p. 8

  • Takeaways & Limitations

    The architecture supports speech interaction while retaining text-LLM abilities such as prompt following and role-playing.

    Source: §1 · p. 2

    Future work must broaden the speech encoder to support emotion understanding and audio captioning, and add more downstream speech-dialogue tasks under a frozen LLM.

    Source: §4 · p. 9

Abstract

from arXiv · show

Rapidly developing large language models (LLMs) have brought tremendous intelligent applications. Especially, the GPT-4o's excellent duplex speech interaction ability has brought impressive experience to users. Researchers have recently proposed several multi-modal LLMs in this direction that can achieve user-agent speech-to-speech conversations. This paper proposes a novel speech-text multimodal LLM architecture called Freeze-Omni. Our main contribution is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process. We design a three-stage training strategy for modeling both the speech input and output, enabling Freeze-Omni to obtain speech-to-speech conversation ability using text-speech paired data (such as ASR and TTS data) and only 60,000 multi-round text Q&A data on 8 GPUs. Moreover, we can effectively ensure that the intelligence of the Freeze-Omni in the speech modality is at the same level compared with that in the text modality of its backbone LLM, while achieving low latency end-to-end spoken response. In addition, we also designed a method to achieve duplex dialogue ability through multi-task training, giving Freeze-Omni a more natural style of dialogue ability between users and agents. In summary, Freeze-Omni holds great potential to conduct speech-to-speech dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting problem caused by limited data and training resources.

1 Introduction

Freeze-Omni addresses the latency and catastrophic-forgetting challenges of speech-to-speech LLMs by connecting speech modalities to a frozen textual LLM. Its staged training uses limited paired and Q&A data while preserving the backbone’s capabilities.

  • Cascaded ASR + LLM + TTS pipelines can impose considerable interaction latency, motivating end-to-end speech interaction.Source: §1 · p. 1
  • Fine-tuning LLMs for speech alignment can cause catastrophic forgetting when large-scale spoken Q&A data is unavailable.Source: §1 · p. 2
  • Freeze-Omni models speech input through ASR alignment and prompt embedding while keeping the LLM frozen.Source: §1 · p. 2
  • Speech output is learned from text-speech pairs with an AR decoder and prefix kv-cache fine-tuning, without updating the LLM.Source: §1 · p. 2
  • A chunk-wise state-prediction task determines whether the user interrupts, enabling duplex speech-to-speech dialogue.Source: §1 · p. 2
  • The LLM remains completely frozen while Freeze-Omni targets low-latency speech-to-speech dialogue and preserves the backbone’s original intelligence.Source: §1 · p. 2
  • The approach uses small-scale paired text-speech Q&A data and fewer computing resources than large spoken-data training would require.Source: §1 · p. 2
  • Freeze-Omni can support text-capable multimodal LLMs while retaining abilities such as prompt following and role-playing.Source: §1 · p. 2

2 Model

Freeze-Omni combines streaming speech encoders and decoders with a frozen backbone LLM through adapters and staged alignment. The architecture processes speech and generated outputs chunk by chunk for streaming interaction.

Overview

  • Freeze-Omni connects a streaming speech encoder and speech decoder to a frozen text-modality LLM to obtain speech-to-speech dialogue.Source: §2.1 · p. 2
  • The architecture sends LLM hidden states and text tokens through chunked NAR prefix and NAR speech decoders, followed by AR decoding and streaming codec synthesis.Source: Figure 1 · p. 3

Speech input

  • Streaming speech is converted into chunk-wise representations, mapped by an adapter into the LLM embedding space, and processed with downsampling for faster response.Source: §2.2.1 · p. 3
  • A three-stage speech-input training strategy teaches speech understanding while keeping the LLM frozen.Source: §2.2.2 · p. 3

Training strategy

  • The final speech-input stage uses generated multi-round answers, multi-speaker TTS questions, and trainable prompt embeddings to align speech conversations with the backbone.Source: §2.2.2 · p. 4

2.3 Modeling of speech output

Freeze-Omni generates speech through a token-based decoder combining NAR semantic modeling, AR token generation, and codec waveform reconstruction. Three training stages align this decoder with the frozen LLM’s outputs.

Architecture

  • The speech decoder combines NAR semantic modeling, AR speech-token generation, and codec decoding into a speech stream.Source: §2.3.1 · p. 5

Training strategy

  • A three-stage training method enables speech generation from LLM outputs while keeping the LLM frozen.Source: §2.3.2 · p. 5
  • Stage one trains a single-codebook codec from speech data, reducing decoder complexity and latency for the limited-speaker setting.Source: §2.3.2 · p. 5
  • Stage two trains NAR and AR decoders on text-speech pairs using frozen LLM embeddings and codec-derived speech-token labels.Source: §2.3.2 · p. 5
  • Stage three trains only the NAR prefix speech decoder on LLM hidden states to couple speech generation to the LLM’s output style.Source: §2.3.2 · p. 6

2.4 Design for duplex dialogue

Freeze-Omni adds chunk-level state prediction to its streaming pipeline so it can detect whether an utterance ends with an interruption. A server strategy schedules chunk processing across idle models.

State prediction

  • State 0 permits continued speech input, while states 1 and 2 mark speech endings and determine whether the LLM should regenerate after interruption.Source: §2.4 · p. 6

Training strategy

  • Multi-task training optimizes both state classification and the LLM to support duplex speech-to-speech dialogue.Source: §2.4 · p. 6

Serving

  • A model-as-a-server setup sends VAD-triggered speech chunks to idle models while separating speech-encoder and LLM caches during inference.Source: §2.4 · p. 6

State prediction

  • An additional classification layer predicts chunk-level states from the LLM hidden state corresponding to each speech chunk’s final frame.Source: Figure 4 · p. 6

3.1 Setups

Freeze-Omni was configured with Qwen2-7B-Instruct as its frozen backbone and trained using paired speech-text data, 60,000 multi-round Q&A examples, and three-stage optimization on 8 GPUs.

Data

  • 60,000 multi-round Q&A examples were used for stage-3 speech-input and speech-output modeling.The examples were generated with new backbone-LLM answers and synthesized speech.Source: §3.1.1 · pp. 6–7
  • 110,000 hours of Chinese and English ASR data supported speech-input modeling across stages 1 and 2.Source: §3.1.1 · p. 6
  • About 3,000 hours of zero-shot-TTS text-speech pairs supported speech-output modeling in stages 1 and 2.Source: §3.1.1 · p. 7

Model configuration

  • Qwen2-7B-Instruct served as the backbone LLM, whose parameters were not updated during training.The setup is intended to verify the method with a public 7B-level LLM.Source: §3.1.2 · p. 7
  • The speech encoder used 4-times convolutional downsampling, 24 transformer layers, a 1024 hidden size, and approximately 350M parameters.Its input was mel-filter bank features with a 25ms window and 10ms shift, and its output frame rate was 12.5Hz.Source: §3.1.2 · p. 7
  • The speech decoder used a single 1024-entry codebook at 40Hz, 4-layer Llama decoders with hidden size 896, and approximately 120M parameters.The codec output sample rate was 24000Hz.Source: §3.1.2 · p. 7

Training

  • All experiments used AdamW with warm-up scheduling and were completed on 8 GPUs.Learning rates differed across training stages, including 2e-4, 1e-4, and 6e-4 for speech-input modeling.Source: §3.1.3 · p. 7

3.2 Results on speech input

Speech-input understanding was evaluated through ASR accuracy after stage 2, with Mandarin measured by CER and English by WER.

Evaluation

  • Mandarin evaluation used CER (%) on aishell-1, test_net, and test_meeting, while English evaluation used WER (%) on dev-clean, dev-other, test-clean, and test-other.Source: §3.2 · p. 7

3.3 Results on speech output

Speech-output quality was assessed on 1,000 utterances using CER under different top-k settings, showing benefits from LLM hidden-state prefix conditioning and an added NAR pre-network.

Evaluation

  • 1,000 utterances were used to compare synthesized-speech ASR accuracy with label text under different AR decoding top-k settings.CER (%) was measured with paraformer-zh.Source: §3.3 · p. 7, Table 2 · p. 8

Prefix conditioning

  • Introducing the LLM hidden state into the NAR prefix speech decoder reduced CER (%) and improved alignment with the LLM.The comparison was between Speech Decoder w/o Prefix and Speech Decoder models.Source: §3.3 · p. 7
  • Increasing top-k produced better robustness for the speech decoder with prefix fine-tuning.Source: §3.3 · p. 7

Decoder architecture

  • An additional two-layer Llama pre-network between the NAR decoder and LLM embedding layer significantly improved generated speech quality.The added parameters address the different spaces represented by LLM embeddings and speech tokens.Source: §3.3 · p. 8

3.4 Results on spoken question answering

Freeze-Omni’s spoken question-answering accuracy was compared with its Qwen2-7B-Instruct backbone on three evaluation sets, showing a smaller accuracy gap than Moshi.

Evaluation

  • Spoken question answering was evaluated on LlaMA-Questions, Web Questions, and Trivia QA.Text-only Web Questions and Trivia QA were synthesized into speech with edge-tts.Source: §3.4 · p. 8

Results

  • Freeze-Omni showed excellent performance, with a smaller accuracy gap from Qwen2-7B-Instruct than Moshi.The authors interpret this as evidence that its intelligence is at the same level across text and speech modalities.Source: §3.4 · p. 8

3.5 Analysis on end-to-end latency

Freeze-Omni’s latency analysis separates statistical latency from manually measured non-statistical latency and reports an average real-world latency of about 1.2 seconds when network effects are included.

Latency definition

  • Statistical latency measures the interval from LLM interruption to the first generated PCM speech chunk, using 40-token speech chunks and sentence-based text segmentation.The analysis divides this interval into four parts.Source: §3.5 · p. 8
  • Non-statistical latency is manually measured because it cannot be counted automatically.The reported non-statistical latency is approximately one to two speech-encoder chunk sizes, or 160–320 ms under the experiment configuration.Source: §3.5 · p. 9

Overall latency

  • About 1.2 seconds is the estimated average latency in real scenarios after including approximately 200–300 ms of network latency.This estimate combines the measured latency analysis with network effects.Source: §3.5 · p. 9

Evaluation context

  • Table 3 evaluates question-answering accuracy across three sets, comparing speech-input and text-input models under zero-shot generation.Freeze-Omni uses Qwen2-7B-Instruct as its backbone, while Moshi uses Helium.Source: Table 3 · p. 9

Latency reporting

  • Table 4 reports statistical latency in milliseconds using median and 90th-percentile values under PyTorch bfloat16 inference.The table’s 50% and 90% columns represent the median and 90th percentile, respectively.Source: Table 4 · p. 9

4 Conclusion and future work

The paper concludes that Freeze-Omni enables low-latency speech-to-speech dialogue without fine-tuning the LLM backbone, while outlining future expansions in audio understanding, speech-dialogue tasks, and speech generation control.

Conclusion

  • Freeze-Omni provides low-latency speech-to-speech dialogue without fine-tuning the LLM backbone and reports strong performance across various tasks.The conclusion presents this as the paper’s central result.Source: §4 · p. 9

Future work

  • The authors plan to upgrade the speech encoder to a general audio encoder for emotion understanding and audio captioning.This would extend the encoder beyond the currently described speech-dialogue setting.Source: §4 · p. 9
  • The authors plan to add more speech-dialogue tasks while keeping the LLM frozen, including capabilities like state prediction.The stated goal is to broaden downstream task coverage under the frozen-LLM condition.Source: §4 · p. 9
  • The speech decoder is planned to support multiple voices and instruction following for richer speaking styles.The planned changes aim to extract more instruction information from the LLM hidden state.Source: §4 · p. 9

Related papers

Loading 2411.00774v5…