Source-linked AI summary
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang
TL;DR
Modular speech systems divide recognition, dialogue, and synthesis into stages, while natural conversation involves listening and speaking at the same time. SALMONN-omni combines streaming speech understanding and generation in a codec-free, full-duplex model with a “thinking” mechanism. The paper reports versatility across speech tasks and simulated handling of turn-taking and barge-in.
Problem
Natural conversation is full-duplex, but modular speech pipelines divide recognition, dialogue, and synthesis into stages, adding complexity and risking error accumulation.
Source: §1 · pp. 1–2Method
Results
The authors report versatility across speech recognition, synthesis, enhancement, dereverberation, target speaker extraction, and spoken question answering, plus effective turn-taking and barge-in handling in simulated experiments.
Source: §1 · p. 2Takeaways & Limitations
The authors present SALMONN-omni as a prototype for modeling diverse interactive spoken-dialogue dynamics with a single end-to-end model.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Full-duplex multimodal large language models (LLMs) provide a unified framework for addressing diverse speech understanding and generation tasks, enabling more natural and seamless human-machine conversations. Unlike traditional modularised conversational AI systems, which separate speech recognition, understanding, and text-to-speech generation into distinct components, multimodal LLMs operate as single end-to-end models. This streamlined design eliminates error propagation across components and fully leverages the rich non-verbal information embedded in input speech signals. We introduce SALMONN-omni, a codec-free, full-duplex speech understanding and generation model capable of simultaneously listening to its own generated speech and background sounds while speaking. To support this capability, we propose a novel duplex spoken dialogue framework incorporating a ``thinking'' mechanism that facilitates asynchronous text and speech generation relying on embeddings instead of codecs (quantized speech and audio tokens). Experimental results demonstrate SALMONN-omni's versatility across a broad range of streaming speech tasks, including speech recognition, speech enhancement, and spoken question answering. Additionally, SALMONN-omni excels at managing turn-taking, barge-in, and echo cancellation scenarios, establishing its potential as a robust prototype for full-duplex conversational AI systems. To the best of our knowledge, SALMONN-omni is the first codec-free model of its kind. A full technical report along with model checkpoints will be released soon.
1 Introduction
Natural conversation involves simultaneous listening, thinking, and speaking, while turn-based systems and modular pipelines do not fully capture that dynamic. SALMONN-omni is presented as a codec-free, end-to-end model for full-duplex speech interaction.
1 Introduction
- Human conversation includes simultaneous listening, thinking, and speaking, with turn-taking, backchanneling, overlapping speech, and barge-in.These dynamics motivate interest in full-duplex speech LLMs.Source: §1 · p. 1
- Traditional modular systems route speech through recognition, dialogue, and synthesis stages, adding complexity and risking error accumulation across subsystems.They typically support turn-based dialogue involving a single speaker.Source: §1 · p. 2
- Earlier codec-based full-duplex models use discrete speech tokens, with Moshi modeling parallel streams and SyncLLM interleaving input and output tokens.The paper also notes that GPT-4o’s underlying technologies remain undisclosed.Source: §1 · p. 2
- SALMONN-omni combines speech understanding and generation in a codec-free, full-duplex end-to-end model for conversational interaction.The paper identifies it as the first model, to the authors’ knowledge, to achieve this without a speech codec.Source: §1 · p. 2
2 Methodology
SALMONN-omni addresses streaming, simultaneous input and output, temporal alignment, and conversational dynamics within one codec-free framework. Its “thinking” strategy supports flexible speech-state transitions while the model continues listening.
2 Methodology
- The end-to-end architecture connects a streaming speech encoder, LLM, and streaming speech synthesizer through embeddings and cross-attention, without explicit text between components.The encoder processes input while the LLM handles output; embeddings link the components.Source: §2 · p. 3, Figure 2 · p. 3
- A periodic synchronization mechanism aligns auditory and textual modalities by processing fixed-duration input and generating a fixed number of textual embeddings per time block.For each block, the encoder extracts auditory embeddings from a fixed duration of speech, and the LLM generates word embeddings conditioned on the current and previous blocks.Source: §2 · p. 3
- The model switches between speaking and non-speaking states with <start_speak> and <end_speak>, continuing to listen during either state.When speaking, generated word embeddings are sent to the synthesizer to produce speech for the same time-block duration; input interruption can trigger a return to non-speaking.Source: §2 · p. 3
- The “thinking” strategy uses <think> as an input in cases where training labels are difficult to collect, without forcing the model to output that token.Outputs are constrained not to include <start_speak> or <end_speak>; the loss assigns thinking tokens a negative coefficient, with state-dependent transition-token labels.Source: §2 · p. 4
3 Case Study
The case study demonstrates SALMONN-omni on streaming speech tasks and simulated conversational interactions, including turn-taking and barge-in.
3 Case Study
- Synthetic data trains SALMONN-omni for turn-taking and barge-in, alongside streaming speech recognition, enhancement, and spoken question answering.Recognition training uses 60k hours of LibriHeavy and 10k hours of GigaSpeech.Source: §3 · p. 4
- SALMONN-omni converts input speech to text in a streaming manner without using the speech synthesizer.Source: Figure 3 · p. 4
- For speech enhancement, SALMONN-omni denoises and dereverberates noisy speech, then re-speaks its content more clearly.Source: Figure 4 · p. 5
- In spoken question answering, SALMONN-omni handles turn-taking.Source: Figure 5 · p. 5
- In context-independent barge-in, the model stops when it hears “Yes” and continues for other words or silence.Source: Figure 6 · p. 6
- With echo cancellation, SALMONN-omni stops for the user’s “Yes,” continues for other words or silence, and remains unaffected by its own speech.Source: Figure 7 · p. 6
- For context-dependent barge-in, SALMONN-omni stops when the user quickly responds but continues speaking when the user says something irrelevant.Source: Figure 8 · p. 7, Figure 9 · p. 7
4 Conclusion
SALMONN-omni combines speech understanding and generation in a codec-free, full-duplex framework. The conclusion describes its broad streaming-task capabilities and simulated conversational behaviors as a prototype for future conversational AI.
4 Conclusion
- SALMONN-omni is a multimodal LLM in a codec-free, full-duplex framework for simultaneous speech understanding and generation.Source: §4 · p. 7
- An end-to-end model and a novel “thinking” strategy unify streaming recognition, enhancement, dereverberation, target speaker extraction, and spoken question answering.Source: §4 · p. 7
- Simulated turn-taking and context-dependent barge-in experiments demonstrate the model’s potential as a prototype for modeling interactive spoken-dialogue dynamics.Source: §4 · p. 7