PaperlayerView exact revision on arXiv

Source-linked AI summary

InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue

Wenwen Tong, Hewei Guo, Dongchuan Ran, Jiangnan Chen, Jiefan Lu, Kaibin Wang, Keqiang Li, Xiaoxu Zhu, Jiakui Li, Kehan Li, Xueheng Li, Lumin Li, Chenxu Guo, Jiasheng Zhou, Jiandong Chen, Xianye Wu, Jiahao Wang, Silei Wu, Lei Chen, Hanming Deng, Yuxuan Song, Dinghao Zhou, Guiping Zhong, Ken Zheng, Shiyin Kang, Lewei Lu

arXiv:2510.13747v2cs.CV

TL;DR

InteractiveOmni addresses the limited support for integrated, long-term audio-visual dialogue in existing multi-modal models. It unifies omni-modal encoders, language processing, and speech generation with multi-stage training and new interaction benchmarks, achieving leading performance across modalities and strong multi-turn capabilities.

  • Problem

    Existing multi-modal models mainly focus on understanding and single-turn interaction, leaving integrated omni-modal input, speech output, and long-term multi-turn ability insufficiently addressed.

  • Method

    InteractiveOmni combines vision and audio encoders, an LLM decoder, and speech generation in one model, trained with omni-modal pre-training, post-training, and curated multi-turn dialogue data.

  • Results

    InteractiveOmni achieves state-of-the-art performance against similarly sized models across image, audio, and video understanding and speech conversation, with InteractiveOmni-4B comparable to Qwen2.5-Omni-7B.

  • Takeaways & Limitations

    The open-source model provides a unified foundation for multi-modal interactive systems with long-term memory and natural speech interaction.

  • Takeaways & Limitations

    Future work must improve efficiency for real-time interaction and the model’s understanding of more complex, abstract inter-modal relationships.

Abstract

from arXiv · show

We introduce InteractiveOmni, a unified and open-source omni-modal large language model for audio-visual multi-turn interaction, ranging from 4B to 8B parameters, designed to lead the field of lightweight models by offering comprehensive omni-modal understanding and speech generation capabilities. To achieve this, we integrate the vision encoder, audio encoder, large language model, and speech decoder into a unified model for understanding and generation tasks. We design a multi-stage training strategy to ensure robust cross-modal capabilities, including pre-training for omni-modal understanding, followed by post-training with speech conversation and audio-visual interaction. To enable human-like long-term conversational ability, we meticulously curate a multi-turn training dataset that enhances the model's ability to handle complex and multi-turn interactions. To effectively evaluate the multi-turn memory and speech interaction capabilities, we construct the multi-modal multi-turn memory benchmark and the multi-turn speech interaction benchmark. Experiments demonstrate that InteractiveOmni significantly outperforms leading open-source models and provides a more intelligent multi-turn audio-visual experience, particularly in its long-term memory capabilities. Notably, InteractiveOmni-4B is comparable to the much larger model like Qwen2.5-Omni-7B on general benchmarks, and it can retain 97% of the performance of the InteractiveOmni-8B while utilizing only 50% of the model size. Achieving state-of-the-art results against similarly sized models across image, audio, video understanding, and speech generation tasks, InteractiveOmni is an accessible, open-source foundation for next-generation intelligent interactive systems.

1 Introduction

InteractiveOmni addresses the gap between omni-modal understanding and human-like multi-turn interaction by unifying perception, language, and speech generation. It combines multi-stage training and dedicated benchmarks to support and evaluate long-term, audio-visual interaction.

  • 1 Introduction: InteractiveOmni targets the limitation that prior multi-modal models mainly emphasize understanding and single-turn interaction rather than long-term multi-modal dialogue.The paper identifies multi-modal alignment, unified streaming generation, memory, empathy, and contextual consistency as central challenges.
  • 1 Introduction: The unified system is designed to perceive audio and video, interact actively, and maintain multi-turn memory and empathy in human-like exchanges.The paper frames holistic integration of vision, hearing, and language as central to natural communication and long-term memory.
  • 1 Introduction: Its training combines omni-modal pre-training with instruction tuning, preference optimization, and curated multi-turn data focused on memory, empathy, and contextual understanding.The authors construct the MMMB and MSIB benchmarks to evaluate multi-turn memory and speech interaction.
  • 1 Introduction: InteractiveOmni achieves state-of-the-art performance against similarly sized models across image, audio, and video understanding and speech conversation tasks.It is also reported to be comparable to larger leading omni-modal models on broad benchmarks.
  • 1 Introduction: The model unifies vision and audio encoders with an LLM and streaming speech decoder, processing visual and audio tokens to generate text and speech sequentially.This architecture supports end-to-end omni-modal perception and generation without separating understanding from response generation.

2 Method

InteractiveOmni unifies omni-modal perception and text-and-speech generation, supported by multi-turn data designed to test historical image and dialogue memory. Its training corpus spans image, video, audio, omni-modal, and speech-interaction data.

  • 2.1 Architecture: InteractiveOmni combines vision and audio encoders, an LLM decoder, speech-token LM, and token2wav generator for end-to-end omni-modal perception and generation.
  • 2.2 Datasets: Generated text question-answer pairs can be converted into speech pairs with TTS to support end-to-end dialogue training.
  • 2.2 Datasets: The multi-turn data pipeline samples visual inputs, generates tailored questions, and creates turns requiring historical image, text, or mixed memory.
  • 2.2 Datasets: The training data includes approximately 12 million image-text pairs, 5 million video-text pairs, over 240,000 audio hours, and approximately 15 million omni-modal data pairs.
  • 2.2 Datasets: The integrated omni-modal data supports comprehensive understanding and memory and multi-turn interactive capabilities.

2.3 Training

InteractiveOmni is trained in progressive pre-training and post-training stages to align modalities, improve instruction following, and enable audio-visual and speech interaction. DPO and model merging further target conversational quality and performance.

  • 2.3 Training: Training first aligns audio, image, video, and text through omni-modal pre-training, then improves instruction following and audio-visual interaction through post-training.
  • 2.3 Training: Pre-training progresses from vision-text alignment to audio-text alignment and finally mixed multi-modal data integration.
  • 2.3 Training: Speech decoding is integrated into the architecture, using aligned text and speech tokens with interleaving for streaming output and hard-sample mining for robustness.
  • 2.3 Training: DPO targets generated-content quality and conversational experience, while model merging is applied during pre-training to enhance performance.

3 Evaluation

The evaluation covers omni-modal understanding and speech generation across in-house multi-turn and open-source benchmarks, comparing InteractiveOmni with proprietary and open-source models.

  • 3 Evaluation: Evaluations span image, video, audio, and text benchmarks plus speech generation, using both in-house multi-turn and open-source datasets.
  • 3 Evaluation: Comparisons include GPT-4o, Gemini, MiniCPM-o-2.6, Qwen2.5-Omni-7B, Kimi-Audio, Qwen2.5-VL, and InternVL3.

3.1 Multi-turn Benchmarks

The MMMB measures historical text and image memory, while MSIB evaluates multi-turn speech interaction across conversational, expressive, and instruction-following dimensions. InteractiveOmni shows strong memory, automated speech-interaction, and human-rated performance against listed baselines.

  • 3.1.1 Multi-modal Multi-turn Memory Benchmark (MMMB): MMMB contains 300 dialogue groups of up to 15 turns and tests text, image, and mixed memory using a final question dependent on historical context.
  • 3.1.1 Multi-modal Multi-turn Memory Benchmark (MMMB): InteractiveOmni-4B outperforms listed vision-language and omni-modal baselines on MMMB, while InteractiveOmni-8B is comparable to Gemini-2.5-Flash at 58.17 versus 60.84.
  • 3.1.1 Multi-modal Multi-turn Memory Benchmark (MMMB): InteractiveOmni-4B maintains 40% accuracy at a turn distance of four as performance declines with increasing recall distance.
  • 3.1.2 Multi-turn Speech Interaction Benchmark (MSIB): InteractiveOmni-4B leads or places second in five of six content categories and exceeds baselines in Emotional Expression and Role-Playing.
  • 3.1.2 Multi-turn Speech Interaction Benchmark (MSIB): InteractiveOmni-4B achieves a 3.95 average MSIB score versus 3.58 for Qwen2.5-Omni-7B and 3.65 for Kimi-Audio, while the 8B model reaches 4.03.
  • 3.1.2 Multi-turn Speech Interaction Benchmark (MSIB): In human evaluation, InteractiveOmni scores higher than Qwen2.5-Omni-7B and Kimi-Audio across five reported conversational dimensions.

3.2 Open-source Benchmarks

InteractiveOmni achieves competitive or leading performance across image, video, audio, omni-modal, speech-question-answering, speech-generation, and spoken-dialogue benchmarks. The results include strong performance from both 4B and 8B variants against similarly sized open-source models.

  • Image Understanding Benchmarks: InteractiveOmni-8B outperforms all open-source models on HallusionBench, achieving a score of 61.3.
  • InteractiveOmni achieves competitive video-understanding performance and nearly state-of-the-art spoken-dialogue performance across four benchmarks.
  • Audio Understanding Benchmarks: InteractiveOmni-8B surpasses open-source omni-modal and audio-language models on WenetSpeech, scoring 5.04 on test-net and 5.55 on test-meeting.
  • InteractiveOmni achieves state-of-the-art performance on OmniBench, with InteractiveOmni-4B attaining an average score of 59.19.
  • Speech Question Answering: InteractiveOmni-4B achieves an average OpenAudioBench score of 69.10, outperforming Kimi-Audio, Step-Audio-chat, and Qwen2.5-Omni-7B.
  • Speech Generation: InteractiveOmni-4B and InteractiveOmni-8B achieve overall WERs of 22.04 and 18.07 on EmergentTTS-Eval, respectively, surpassing all other models.

4 Related works

Related work spans vision-language, audio-language, speech-to-speech, and omni-modal models, while highlighting unresolved challenges in unified multi-modal interaction. Existing systems often emphasize understanding or rely on modality-specific representations and cascaded pipelines.

  • Vision-language and audio-language models support cross-modal understanding, but prior work mainly targets single-turn interaction rather than long-term multi-modal dialogue.
  • Speech-to-speech systems commonly cascade recognition, language modeling, and synthesis, creating latency, paralinguistic-perception, and error-propagation constraints.
  • Prior speech systems explore interleaved data, multi-stage training, multi-codebook discretization, and hybrid text-audio representations for end-to-end spoken dialogue.
  • Omni-modal models integrate image, video, audio, and text to capture richer contextual relationships than models restricted to fewer modalities.
  • Discrete audio-token approaches expand the language-model vocabulary, whereas continuous audio-feature approaches provide an alternative representation for unified understanding and generation.

5 Conclusions

InteractiveOmni unifies multi-modal understanding and speech generation through a single framework, supported by staged training and curated multi-turn dialogue data. It achieves strong cross-task and multi-turn results, while future work targets efficiency and more complex inter-modal reasoning.

  • InteractiveOmni integrates text, image, audio, and video processing with direct generation of coherent text and speech.
  • Omni-modal pre-training, post-training, and curated multi-turn dialogue data support cross-modal synergy, contextual awareness, and long-term memory.
  • InteractiveOmni achieves state-of-the-art performance against similarly sized models across image, audio, and video understanding and speech conversation benchmarks.
  • Future work will improve real-time interaction efficiency and expand comprehension of more complex, abstract inter-modal relationships.

A.1 Data Preparation and Inference Pipeline

The multi-turn speech interaction benchmark evaluates speech-to-speech dialogue using generated and manually revised conversations converted into speech. Evaluation focuses on the final turn while preserving earlier turns as conversational history.

  • The benchmark contains 244 dialogues spanning 2 to 10 turns and six dimensions, including emotion, speech-rate control, role playing, creativity, and instruction following.
  • Text-based Dialogue Construction: Text-based dialogues are generated with an LLM and manually revised, with only the final turn evaluated against the predefined conversational history.
  • Speech-based Dialogue Construction: A high-quality TTS system converts textual multi-turn prompts into speech to create speech-based dialogue evaluations.

A.2 Human Evaluation

Human evaluators rate end-to-end speech interaction on a 1–5 MOS scale, separately considering speech quality and content/attribute fulfillment. The rubric first categorizes outputs by whether content and attributes are good, then assigns detailed scores.

  • A.2 Human Evaluation: Expert evaluators rate generated speech on a 1–5 Mean Opinion Score scale covering speech content and speech quality.The evaluation uses a detailed scoring rubric.
  • A.2 Human Evaluation: Speech-quality scores range from unintelligible or robotic speech at 1–2 to human-like, expressive, and clear speech at 4–5.Score 3 represents clear human-sounding speech with limited emotion or noticeable flaws.
  • A.2 Human Evaluation: Outputs with good content and attributes proceed to the 4–5 range, where score 4 denotes satisfactory completion and score 5 denotes precise, highly effective fulfillment.Score 5 may include extra value, vivid role-play, or exceptionally strong instruction following.

A.3 Automated Machine Evaluation.

The automated evaluation pipeline uses Gemini-2.5-Pro as an LLM judge to transcribe generated audio and score speech and content quality. It evaluates speech clarity, naturalness, continuity, content matching, and attribute matching using 1–5 rubrics and structured JSON output.

  • A.3 Automated Machine Evaluation.: Gemini-2.5-Pro serves as the automated judge for scalable end-to-end speech-interaction assessment.The model is selected for multimodal understanding and complex reasoning capabilities.
  • A.3 Automated Machine Evaluation.: Speech quality is assessed through clarity, naturalness, and continuity, including noise, human-likeness, emotional expressiveness, interruptions, stutters, and glitches.These criteria are scored on a 1–5 scale.
  • A.3 Automated Machine Evaluation.: Content quality measures whether the transcript matches key reference information and whether emotion, speed, role, and other attributes fit the expected context.Attribute matching considers the conversation history.
  • A.3 Automated Machine Evaluation.: The judge first transcribes the audio, then evaluates speech and content quality using the specified rubric.The pipeline therefore bases evaluation on both recognized speech content and the generated audio.
  • A.3 Automated Machine Evaluation.: The speech-quality rubric ranges from unintelligible or robotic output at 1–2 to clear, expressive, human-like output at 4–5, with score 3 indicating noticeable limitations.The content rubric uses a two-step category and detailed-scoring procedure.
  • A.3 Automated Machine Evaluation.: The judge returns a JSON object containing the transcript, separate speech- and content-quality scores, and brief reasoning for each score.The format also includes background text and audio as evaluation inputs.
Loading 2510.13747v2…