Source-linked AI summary
VITA: Towards Open-Source Interactive Omni Multimodal LLM
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, Haoyu Cao, Di Yin, Long Ma, Xiawu Zheng, Rongrong Ji, Yunsheng Wu, Ran He, Caifeng Shan, Xing Sun
TL;DR
VITA addresses the shortage of open-source models combining unified multimodal understanding with natural interaction. It extends Mixtral 8x7B through bilingual and multimodal staged training, and reports strong multimodal performance alongside non-awakening and audio-interrupt interaction. The authors position it as an open-source starting point for integrating these capabilities, while noting remaining gaps to proprietary models and noisy-audio misclassification.
Problem
Open-source models rarely combine end-to-end processing of video, image, text, and audio with natural multimodal human-computer interaction.
Method
VITA expands Mixtral 8x7B’s Chinese vocabulary, applies bilingual instruction tuning, aligns visual and audio encoders, and performs multimodal instruction tuning with state tokens.
Results
VITA shows comparable image understanding to LLaVA-Next, approaches Gemini 1.5 Pro, and provides non-awakening and audio-interrupt interaction.
Takeaways & Limitations
VITA provides an open-source unified framework for video, image, text, and audio understanding with preliminary multimodal interaction capabilities.
Takeaways & Limitations
VITA still has a notable performance gap relative to proprietary counterparts and sometimes misclassifies noisy audio as query audio.
Abstract
from arXiv · showhide
The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research. Project Page: https://vita-home.github.io.
1 Introduction
VITA addresses the lack of open-source models that combine end-to-end multimodal processing with natural multimodal interaction. It integrates video, image, text, and audio capabilities through staged training and introduces non-awakening and audio-interrupt interaction.
- Open-source models have generally lacked GPT-4o’s combination of unified text, vision, and audio processing and natural multimodal interaction.
- VITA’s pipeline expands Mixtral 8×7B’s Chinese capabilities, aligns modality encoders with the LLM, and applies multimodal instruction tuning.
- A duplex deployment runs generation and monitoring models concurrently, allowing a new effective audio query to pause the current response and receive immediate handling.
- VITA supports video, image, text, and audio inputs in English and Chinese, either as pure text/audio or multimodal combinations.
- State tokens identify input-query types, enabling non-awakening interaction that filters noisy audio without a wake-up word or button.
- The authors open-source the model, training code, and inference deployment framework to support further community development.
2 Related Work
Prior open-source MLLMs have advanced multimodal understanding, but most remain focused on image-text inputs and rarely emphasize natural user interaction. VITA’s training pipeline targets this broader integration.
- Recent open-source MLLMs have improved vision-language alignment, instruction following, and other multimodal foundational capabilities.
- Most open-source models focus on image-text modalities, while proprietary systems support broader combinations including audio, image, and text.
- Open-source MLLMs have comparatively underexplored user interaction capabilities, motivating VITA’s multimodal interaction focus.
3 VITA
VITA’s overall development uses a three-stage pipeline: LLM instruction tuning, multimodal alignment, and multimodal instruction tuning.
- VITA’s training pipeline consists of LLM instruction tuning, multimodal alignment, and multimodal instruction tuning.
3.1 LLM Instruction Tuning
The authors adapt Mixtral 8x7B for bilingual Chinese-English instruction following by expanding its vocabulary and training on synthetic bilingual text.
- The vocabulary expands from 32,000 to 51,747 tokens, followed by instruction tuning on 5 million synthetic bilingual examples.
3.2 Multimodal Alignment
VITA aligns visual and audio modalities with Mixtral through modality-specific encoders, connectors, and multimodal training data. The alignment design also standardizes training inputs to support longer contexts and improved computational efficiency.
- VITA bridges representation gaps between text and other modalities to establish multimodal understanding.
- Visual Modality: The visual pipeline uses InternViT-300M-448px, dynamic patching for high-resolution images, and frame sampling rules for videos.Videos shorter than 4 seconds use 4 frames, videos from 4 to 16 seconds use one frame per second, and longer videos use 16 frames.
- Visual Modality: Visual alignment trains only the visual connector using image and video descriptions, question answering, and multimodal datasets.Pure text data is reserved for multimodal instruction tuning because the language-model parameters are frozen during visual alignment.
- Data Concatenation: Pure text and image data are concatenated toward 6K-token contexts, whereas video data is sampled frame by frame without concatenation.Concatenation supports multiple image-question interactions and balances token counts across training batches.
- Audio Modality: The audio encoder applies mel filtering, CNN downsampling, and 24 transformer layers, encoding each 2 seconds of audio into 25 tokens.A two-layer MLP connects the resulting audio representation to the text modality.
- Audio Modality: Audio alignment combines Chinese- and English-focused ASR data with audio-captioning data.WenetSpeech and GigaSpeech each provide about 10,000 hours of speech data, while AudioSet SL from WavCaps provides 400K clips with captions.
3.3 Multimodal Instruction Tuning
Multimodal instruction tuning improves instruction following across text and audio while teaching VITA to distinguish query modalities and non-query audio. State tokens condition the model’s response behavior for different interaction types.
- 3.3 Multimodal Instruction Tuning: Instruction tuning enhances instruction following for both text and audio inputs.
- 3.3.1 Training Data: About half of the questions are replaced with audio versions generated using TTS to strengthen audio-query understanding.Different system prompts distinguish image, video, and pure-text data types.
- 3.3.1 Training Data: 474K answer sentences are sampled as noisy-audio examples representing inputs that do not require a user response.These negative samples target non-query-related content for selective response behavior.
- 3.3.2 Training Process: The model distinguishes query audio, noisy audio, and query text using three corresponding state tokens.The tokens are inserted at the beginning of answers during training.
- 3.3.2 Training Process: State token <1> marks audio-initiated questions whose answers are presented as text or speech through TTS.
- 3.3.2 Training Process: State token <2> marks noisy audio and serves as an inference-time special EOS token.Training uses the LLM’s text response as the target because abruptly terminating outputs degrades performance.
- 3.3.2 Training Process: State token <3> marks pure-text questions, completing the three-way query distinction.
- 3.3.2 Training Process: Visual and audio encoders remain frozen while their connectors are trained with Mixtral 8×7B.
3.4 Development with Duplex Pipeline
VITA’s duplex deployment supports non-awakening interaction by filtering environmental audio and supports interruption by monitoring new queries during generation. A separate monitoring model can stop generation when it detects a query.
- Non-awakening Interaction: Non-awakening interaction activates VITA for user audio questions without a wake-up word or button.
- Non-awakening Interaction: Real-time environmental sound tracking determines whether incoming audio contains human speech.Voice Activity Detection is used for this requirement.
- Non-awakening Interaction: The system filters noisy audio so that it responds only to effective human queries.SileroVAD is used for speech detection, while state token <2> distinguishes effective queries from non-query audio.
- Non-awakening Interaction: Non-query audio causes inference to terminate directly, limiting responses to query-type inputs.
- Audio Interrupt Interaction: Audio interrupt interaction tracks and filters external queries while the model is generating a response.
- Audio Interrupt Interaction: When a new question appears, the system stops current generation, consolidates historical context, and answers the new query.
- Audio Interrupt Interaction: The duplex framework deploys concurrent Generation and Monitoring VITA models, with the latter detecting query audio and interrupting the former.The Monitoring model disregards noisy audio and stops the Generation model when it identifies query audio.
4 Evaluation
VITA shows strong performance across language, audio, image, and video evaluations, matching leading open-source models in multimodal understanding while remaining behind proprietary counterparts.
- Language Performance: VITA’s training significantly improves Chinese evaluation performance, maintains English benchmark performance, and improves mathematical reasoning.The evaluated tasks include C-EVAL, AGIEVAL, MMLU, and GSM8K across Chinese and English contexts.
- Audio Performance: VITA achieves considerable results on ASR benchmarks spanning easier aligned data, more challenging meeting data, and unseen-dataset generalization.WenetSpeech includes test_net and test_meeting splits, while Librispeech evaluates generalization on unseen data.
- Multimodal Performance: VITA shows image understanding comparable to LLaVA-Next and close to Gemini 1.5 Pro.The evaluation covers ten representative multimodal benchmarks, including image and video tasks.
- Multimodal Performance: In video understanding, VITA has a small gap relative to LLaVA-Next-Video, while current open-source models retain a substantial gap from proprietary models.The paper frames VITA’s broader modality support and interaction focus as relevant context for the video comparison.
5 Conclusion and Future Work
VITA unifies multimodal understanding with interactive capabilities, but the authors identify remaining gaps in foundational performance, noisy-audio handling, and real-time speech generation.
- Conclusion: VITA integrates video, image, text, and audio understanding with non-awakening and audio-interrupt interaction.The paper presents these capabilities as preliminary open-source explorations of natural multimodal interaction.
- Limitations: VITA remains behind proprietary counterparts despite competitive unimodal and multimodal performance relative to leading open-source models.The limitation concerns enhancement of foundational capabilities.
- Limitations: VITA sometimes misclassifies noisy audio as query audio, motivating more nuanced noisy-audio construction.The current construction samples non-query responses from existing data as noisy-audio samples.
- Limitations: An external TTS tool converts generated text into speech, adding substantial time and limiting real-time interaction efficiency.The authors suggest combining TTS with the LLM for end-to-end speech output.