Source-linked AI summary
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, Long Ma
TL;DR
Freeze-Omni addresses the difficulty of adding speech interaction to LLMs without large spoken datasets or catastrophic forgetting. It connects streaming speech modules to a frozen textual LLM through staged training, retaining the backbone’s capabilities while supporting low-latency speech-to-speech dialogue. The paper reports strong spoken-question-answering performance relative to the backbone and introduces duplex interruption handling, while identifying broader speech and downstream-task support as future work.
Problem
Existing speech-text multimodal LLMs often fine-tune their backbones, while collecting million-hour-scale spoken Q&A data is difficult and can cause catastrophic forgetting.
Method
Freeze-Omni connects streaming speech input and output modules to a frozen text-modality LLM using three-stage training and chunk-level state prediction.
Results
Freeze-Omni preserves the backbone’s intelligence in spoken question answering while achieving low-latency speech-to-speech dialogue with the LLM completely frozen.
Takeaways & Limitations
The architecture supports speech interaction while retaining text-LLM abilities such as prompt following and role-playing.
Takeaways & Limitations
Future work must broaden the speech encoder to support emotion understanding and audio captioning, and add more downstream speech-dialogue tasks under a frozen LLM.
Abstract
from arXiv · showhide
Rapidly developing large language models (LLMs) have brought tremendous intelligent applications. Especially, the GPT-4o's excellent duplex speech interaction ability has brought impressive experience to users. Researchers have recently proposed several multi-modal LLMs in this direction that can achieve user-agent speech-to-speech conversations. This paper proposes a novel speech-text multimodal LLM architecture called Freeze-Omni. Our main contribution is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process. We design a three-stage training strategy for modeling both the speech input and output, enabling Freeze-Omni to obtain speech-to-speech conversation ability using text-speech paired data (such as ASR and TTS data) and only 60,000 multi-round text Q&A data on 8 GPUs. Moreover, we can effectively ensure that the intelligence of the Freeze-Omni in the speech modality is at the same level compared with that in the text modality of its backbone LLM, while achieving low latency end-to-end spoken response. In addition, we also designed a method to achieve duplex dialogue ability through multi-task training, giving Freeze-Omni a more natural style of dialogue ability between users and agents. In summary, Freeze-Omni holds great potential to conduct speech-to-speech dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting problem caused by limited data and training resources.
1 Introduction
Freeze-Omni addresses the latency and catastrophic-forgetting challenges of speech-to-speech LLMs by connecting speech modalities to a frozen textual LLM. Its staged training uses limited paired and Q&A data while preserving the backbone’s capabilities.
- Cascaded ASR + LLM + TTS pipelines can impose considerable interaction latency, motivating end-to-end speech interaction.
- Fine-tuning LLMs for speech alignment can cause catastrophic forgetting when large-scale spoken Q&A data is unavailable.
- Freeze-Omni models speech input through ASR alignment and prompt embedding while keeping the LLM frozen.
- Speech output is learned from text-speech pairs with an AR decoder and prefix kv-cache fine-tuning, without updating the LLM.
- A chunk-wise state-prediction task determines whether the user interrupts, enabling duplex speech-to-speech dialogue.
- The LLM remains completely frozen while Freeze-Omni targets low-latency speech-to-speech dialogue and preserves the backbone’s original intelligence.
- The approach uses small-scale paired text-speech Q&A data and fewer computing resources than large spoken-data training would require.
- Freeze-Omni can support text-capable multimodal LLMs while retaining abilities such as prompt following and role-playing.
2 Model
Freeze-Omni combines streaming speech encoders and decoders with a frozen backbone LLM through adapters and staged alignment. The architecture processes speech and generated outputs chunk by chunk for streaming interaction.
- Overview: Freeze-Omni connects a streaming speech encoder and speech decoder to a frozen text-modality LLM to obtain speech-to-speech dialogue.
- Overview: The architecture sends LLM hidden states and text tokens through chunked NAR prefix and NAR speech decoders, followed by AR decoding and streaming codec synthesis.
- Speech input: Streaming speech is converted into chunk-wise representations, mapped by an adapter into the LLM embedding space, and processed with downsampling for faster response.
- Speech input: A three-stage speech-input training strategy teaches speech understanding while keeping the LLM frozen.
- Training strategy: The final speech-input stage uses generated multi-round answers, multi-speaker TTS questions, and trainable prompt embeddings to align speech conversations with the backbone.
2.3 Modeling of speech output
Freeze-Omni generates speech through a token-based decoder combining NAR semantic modeling, AR token generation, and codec waveform reconstruction. Three training stages align this decoder with the frozen LLM’s outputs.
- Architecture: The speech decoder combines NAR semantic modeling, AR speech-token generation, and codec decoding into a speech stream.
- Training strategy: A three-stage training method enables speech generation from LLM outputs while keeping the LLM frozen.
- Training strategy: Stage one trains a single-codebook codec from speech data, reducing decoder complexity and latency for the limited-speaker setting.
- Training strategy: Stage two trains NAR and AR decoders on text-speech pairs using frozen LLM embeddings and codec-derived speech-token labels.
- Training strategy: Stage three trains only the NAR prefix speech decoder on LLM hidden states to couple speech generation to the LLM’s output style.
2.4 Design for duplex dialogue
Freeze-Omni adds chunk-level state prediction to its streaming pipeline so it can detect whether an utterance ends with an interruption. A server strategy schedules chunk processing across idle models.
- State prediction: State 0 permits continued speech input, while states 1 and 2 mark speech endings and determine whether the LLM should regenerate after interruption.
- Training strategy: Multi-task training optimizes both state classification and the LLM to support duplex speech-to-speech dialogue.
- Serving: A model-as-a-server setup sends VAD-triggered speech chunks to idle models while separating speech-encoder and LLM caches during inference.
- State prediction: An additional classification layer predicts chunk-level states from the LLM hidden state corresponding to each speech chunk’s final frame.
3.1 Setups
Freeze-Omni was configured with Qwen2-7B-Instruct as its frozen backbone and trained using paired speech-text data, 60,000 multi-round Q&A examples, and three-stage optimization on 8 GPUs.
- Data: 60,000 multi-round Q&A examples were used for stage-3 speech-input and speech-output modeling.The examples were generated with new backbone-LLM answers and synthesized speech.
- Data: 110,000 hours of Chinese and English ASR data supported speech-input modeling across stages 1 and 2.
- Data: About 3,000 hours of zero-shot-TTS text-speech pairs supported speech-output modeling in stages 1 and 2.
- Model configuration: Qwen2-7B-Instruct served as the backbone LLM, whose parameters were not updated during training.The setup is intended to verify the method with a public 7B-level LLM.
- Model configuration: The speech encoder used 4-times convolutional downsampling, 24 transformer layers, a 1024 hidden size, and approximately 350M parameters.Its input was mel-filter bank features with a 25ms window and 10ms shift, and its output frame rate was 12.5Hz.
- Model configuration: The speech decoder used a single 1024-entry codebook at 40Hz, 4-layer Llama decoders with hidden size 896, and approximately 120M parameters.The codec output sample rate was 24000Hz.
- Training: All experiments used AdamW with warm-up scheduling and were completed on 8 GPUs.Learning rates differed across training stages, including 2e-4, 1e-4, and 6e-4 for speech-input modeling.
3.2 Results on speech input
Speech-input understanding was evaluated through ASR accuracy after stage 2, with Mandarin measured by CER and English by WER.
- Evaluation: Mandarin evaluation used CER (%) on aishell-1, test_net, and test_meeting, while English evaluation used WER (%) on dev-clean, dev-other, test-clean, and test-other.
3.3 Results on speech output
Speech-output quality was assessed on 1,000 utterances using CER under different top-k settings, showing benefits from LLM hidden-state prefix conditioning and an added NAR pre-network.
- Evaluation: 1,000 utterances were used to compare synthesized-speech ASR accuracy with label text under different AR decoding top-k settings.CER (%) was measured with paraformer-zh.
- Prefix conditioning: Introducing the LLM hidden state into the NAR prefix speech decoder reduced CER (%) and improved alignment with the LLM.The comparison was between Speech Decoder w/o Prefix and Speech Decoder models.
- Prefix conditioning: Increasing top-k produced better robustness for the speech decoder with prefix fine-tuning.
- Decoder architecture: An additional two-layer Llama pre-network between the NAR decoder and LLM embedding layer significantly improved generated speech quality.The added parameters address the different spaces represented by LLM embeddings and speech tokens.
3.4 Results on spoken question answering
Freeze-Omni’s spoken question-answering accuracy was compared with its Qwen2-7B-Instruct backbone on three evaluation sets, showing a smaller accuracy gap than Moshi.
- Evaluation: Spoken question answering was evaluated on LlaMA-Questions, Web Questions, and Trivia QA.Text-only Web Questions and Trivia QA were synthesized into speech with edge-tts.
- Results: Freeze-Omni showed excellent performance, with a smaller accuracy gap from Qwen2-7B-Instruct than Moshi.The authors interpret this as evidence that its intelligence is at the same level across text and speech modalities.
3.5 Analysis on end-to-end latency
Freeze-Omni’s latency analysis separates statistical latency from manually measured non-statistical latency and reports an average real-world latency of about 1.2 seconds when network effects are included.
- Latency definition: Statistical latency measures the interval from LLM interruption to the first generated PCM speech chunk, using 40-token speech chunks and sentence-based text segmentation.The analysis divides this interval into four parts.
- Latency definition: Non-statistical latency is manually measured because it cannot be counted automatically.The reported non-statistical latency is approximately one to two speech-encoder chunk sizes, or 160–320 ms under the experiment configuration.
- Overall latency: About 1.2 seconds is the estimated average latency in real scenarios after including approximately 200–300 ms of network latency.This estimate combines the measured latency analysis with network effects.
- Evaluation context: Table 3 evaluates question-answering accuracy across three sets, comparing speech-input and text-input models under zero-shot generation.Freeze-Omni uses Qwen2-7B-Instruct as its backbone, while Moshi uses Helium.
- Latency reporting: Table 4 reports statistical latency in milliseconds using median and 90th-percentile values under PyTorch bfloat16 inference.The table’s 50% and 90% columns represent the median and 90th percentile, respectively.
4 Conclusion and future work
The paper concludes that Freeze-Omni enables low-latency speech-to-speech dialogue without fine-tuning the LLM backbone, while outlining future expansions in audio understanding, speech-dialogue tasks, and speech generation control.
- Conclusion: Freeze-Omni provides low-latency speech-to-speech dialogue without fine-tuning the LLM backbone and reports strong performance across various tasks.The conclusion presents this as the paper’s central result.
- Future work: The authors plan to upgrade the speech encoder to a general audio encoder for emotion understanding and audio captioning.This would extend the encoder beyond the currently described speech-dialogue setting.
- Future work: The authors plan to add more speech-dialogue tasks while keeping the LLM frozen, including capabilities like state prediction.The stated goal is to broaden downstream task coverage under the frozen-LLM condition.
- Future work: The speech decoder is planned to support multiple voices and instruction following for richer speaking styles.The planned changes aim to extract more instruction information from the LLM hidden state.