Source-linked AI summary
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, Ran He
TL;DR
MLLMs have emphasized vision and text, while jointly achieving strong vision and speech remains difficult because the modalities create training conflicts and conventional speech pipelines add latency. VITA-1.5 progressively trains vision, audio input, and end-to-end audio output, achieving competitive multimodal benchmark performance and efficient speech-to-speech interaction. The model remains generic and does not incorporate individual preferences, and long interactions motivate future long-term-memory support.
Problem
MLLMs have focused mainly on vision and text, while integrating vision and speech is difficult because modality differences create training conflicts and separate ASR/TTS modules can increase latency.
Method
VITA-1.5 uses three-stage progressive training and an end-to-end speech output architecture to integrate vision, language, and speech.
Results
VITA-1.5 performs competitively across multimodal benchmarks, with comparable visual perception and reasoning, leading Mandarin and English ASR accuracy, and significant speech improvements.
Takeaways & Limitations
VITA-1.5 supports efficient speech-to-speech multimodal interaction without separate ASR or TTS modules and enables near-real-time vision-and-speech interaction.
Takeaways & Limitations
VITA-1.5 is generic and does not incorporate individual preferences during interaction; long-term memory is also identified as important for extended interactions.
Abstract
from arXiv · showhide
Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a significant challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing our method against state-of-the-art counterparts across benchmarks for image, video, and speech tasks, we demonstrate that our model is equipped with both strong visual and speech capabilities, making near real-time vision and speech interaction. Code has been released at https://github.com/VITA-MLLM/VITA.
1 Introduction
VITA-1.5 addresses challenges in combining vision and speech by progressively training a multimodal LLM, targeting fluent near-real-time interaction while preserving visual capabilities.
- Speech has become important for multimodal dialogue because it improves information transmission, naturalness, and interaction convenience.
- Visual and speech modalities differ fundamentally, creating training conflicts that can cause one modality’s performance to degrade when the other is added.
- The end-to-end design supports fluent speech replies without separate ASR and TTS modules, addressing latency and coherence limitations of traditional speech-to-speech systems.
- VITA-1.5 uses three training stages that progressively incorporate vision, audio input, and audio output data.The stages establish vision-language ability, add speech understanding, and train end-to-end speech generation without external TTS modules.
- Across image, video, and speech benchmarks, VITA-1.5 shows comparable perception and reasoning to leading image/video MLLMs and significant speech improvements.
2 Related Work
Prior work has advanced visual and speech capabilities, but open-source systems still trail proprietary multimodal models and often rely on latency-inducing speech pipelines.
- Multimodal research has extended language models toward visual understanding and reasoning through multimodal alignment and instruction tuning.
- Speech-enabled systems commonly cascade ASR, LLM, and TTS modules, which can lose paralinguistic features and introduce latency.
- Most open-source models focus primarily on image and text, while multimodal interaction involving audio remains relatively unexplored.
- VITA-1.5 targets this gap with refined training, four-modality perception, and near-real-time vision-and-speech interaction.
3.1 Model Architecture
VITA-1.5 combines vision and audio encoders with adapters and an LLM, then produces speech through an integrated output stack instead of an external TTS model.
- The input architecture combines Vision/Audio Transformers and Multi-Layer Connectors with an LLM for unified vision, language, and audio understanding.
- The output side uses an end-to-end speech module rather than the external TTS model used by VITA-1.0.
- The visual encoder generates 256 visual tokens per 448×448 image, and dynamic patching captures local details in high-resolution images.
- Videos are processed as multiple-image inputs with duration-dependent frame sampling, while dynamic patching is omitted to preserve processing efficiency.
- The audio encoder uses downsampling and Transformer blocks to reduce audio-feature frame rates before LLM processing.It outputs audio features at 12.5Hz.
- Speech generation uses NAR and AR decoders before a codec converts speech tokens into a continuous waveform.
3.2 Training Data
VITA-1.5’s multimodal instruction-tuning data span image, video, text, and speech resources, with different subsets selected for different training objectives.
- The instruction-tuning data include caption and question-answer categories in both Chinese and English.
- Image data cover captioning, visual question answering, OCR, diagrams, and mathematical or visual reasoning tasks.
- Video and pure-text datasets support video captioning, video question answering, language understanding, and text-based question answering.
- 110,000 hours of Chinese-English speech-transcription pairs train and align the audio encoder with the LLM.
- 3,000 hours of TTS-generated text-speech pairs train the speech decoder.
3.3 Three Stage Training Strategies
VITA-1.5 progressively incorporates vision, audio input, and audio output through three training stages designed to reduce modality conflicts. The stages build visual understanding first, then speech understanding and speech generation.
- The three-stage strategy gradually introduces vision and audio into the LLM to relieve conflicts between modalities.
- Stage 1: Vision-Language Training: Stage 1 trains vision-language capabilities through alignment, image understanding, and visual instruction tuning.It uses caption data for alignment and understanding, then combines QA data with retained caption data for visual question answering.
- Stage 2: Audio Input Tuning: Stage 2 adds audio input through speech alignment and speech QA instruction tuning.Audio alignment uses 11,000 hours of speech-transcription pairs, while speech QA samples caption and QA data and includes speech versions of text questions.
- Stage 3: Audio Output Tuning: Stage 3 introduces speech output using 3,000 hours of text-speech data, codec training, and NAR plus AR speech decoder training.The codec maps speech to discrete tokens and back, while the decoders generate speech tokens from text-derived representations.
4 Evaluation
VITA-1.5 is evaluated against open- and closed-source multimodal models on image, video, and speech benchmarks. It retains near-original visual-language performance, remains competitive on vision tasks, and achieves leading Mandarin and English ASR accuracy.
- VITA-1.5 is compared with numerous open-source and closed-source MLLMs across image, video, and speech evaluations.
- Vision-Language Evaluation: VITA-1.5 performs comparably to many open-source models on video understanding but still lags behind GPT-4o and Gemini-1.5-Pro.
- Vision-Language Evaluation: VITA-1.5 performs comparably to leading open-source image models and some closed-source models while retaining almost its original visual-language capabilities after audio training.
- Speech Evaluation: VITA-1.5 achieves leading accuracy on Mandarin and English ASR tasks.The evaluations use Character Error Rate for Mandarin and Word Error Rate for English; the model outperforms specialized speech models in both languages.
5 Conclusion and Future Work
The paper presents VITA-1.5 as a three-stage, end-to-end multimodal LLM for robust vision and speech understanding and efficient speech-to-speech interaction. Future work targets personalization and long-term memory for extended interactions.
- VITA-1.5 integrates vision and speech through a three-stage training strategy and supports speech-to-speech interaction without separate ASR or TTS modules.
- Extensive evaluations show competitive performance across multimodal benchmarks.
- Future Work: VITA-1.5 is currently generic and does not incorporate individual preferences during interaction.
- Future Work: Long-term memory is identified as important for human-computer interactions lasting 10 minutes or several hours.
Impact Statement
The impact statement characterizes the paper as research on large-model technology intended to enhance technical capabilities.
- The paper studies large-model technology to enhance its technical level.