Source-linked AI summary
Ola: Pushing the Frontiers of Omni-Modal Language Model
Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, Yongming Rao
TL;DR
Ola addresses the performance gap between open omni-modal models and specialized systems by combining architectural improvements, curated multimodal data, and cross-modal training. It uses video as a bridge among modalities and progressive alignment, achieving competitive image, video, and audio understanding with 7B parameters. The model’s scope is limited to text generation and does not include audio generation.
Problem
Existing open omni-modal models lag behind specialized LLMs while trying to maintain balanced performance across modalities.
Method
Ola combines extensible modality encoders, unified token alignment, curated multimodal data, video-centered cross-modal learning, and progressive modality alignment.
Results
Ola achieves competitive performance across image, video, and audio understanding, including 68.4% on VideoMME, 3.1 mean WER on LibriSpeech, and 6.41 on AIR-Bench.
Takeaways & Limitations
Ola provides an open omni-modal understanding solution that connects modalities through joint learning and progressive alignment.
Takeaways & Limitations
Ola focuses on text generation and does not explore audio generation, limiting applicability where audio generation is crucial.
Abstract
from arXiv · showhide
Recent advances in large language models, particularly following GPT-4o, have sparked increasing interest in developing omni-modal models capable of understanding more modalities. While some open-source alternatives have emerged, there is still a notable lag behind specialized single-modality models in performance. In this paper, we present Ola, an Omni-modal Language model that achieves competitive performance across image, video, and audio understanding compared to specialized counterparts, pushing the frontiers of the omni-modal language model to a large extent. We conduct a comprehensive exploration of architectural design, data curation, and training strategies essential for building a robust omni-modal model. Ola incorporates advanced visual understanding and audio recognition capabilities through several critical and effective improvements over mainstream baselines. Moreover, we rethink inter-modal relationships during omni-modal training, emphasizing cross-modal alignment with video as a central bridge, and propose a progressive training pipeline that begins with the most distinct modalities and gradually moves towards closer modality alignment. Extensive experiments demonstrate that Ola surpasses existing open omni-modal LLMs across all modalities while achieving highly competitive performance compared to state-of-the-art specialized models of similar sizes. We aim to make Ola a fully open omni-modal understanding solution to advance future research in this emerging field. Model weights, code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
1 Tsinghua University 2 Tencent Hunyuan Research 3 S-Lab, NTU
Figure 1 presents Ola’s visual and audio innovations, while Figure 2 compares its omni-modal performance with other models.
- Ola’s innovations target visual and audio understanding within an omni-modal language model.
- The paper acknowledges equal contribution among marked authors.
- Figure 2 compares Ola with open-source multimodal, omni-modal, and GPT-4o models across image, video, and audio benchmarks.
1. Introduction
The paper addresses the performance gap between open omni-modal models and specialized systems by developing Ola’s architecture, data, and modality-alignment strategy. Ola uses video to connect modalities and reports competitive image, video, and audio results with 7B parameters.
- Existing open omni-modal models retain a substantial performance gap from specialized LLMs, limiting their progress toward real-world applications.
- Ola combines an extensible omni-modal architecture, visual and audio encoders, joint alignment, and curated multimodal data.
- Video serves as the central bridge because visual content, audio, and subtitles are highly correlated within samples.
- With 7B parameters, Ola achieves 68.4% accuracy on VideoMME using video and audio inputs.
- Ola reports 72.6% mean accuracy on OpenCompass, 3.1 mean WER on LibriSpeech, and 6.41 GPT-eval score on AIR-Bench.
2. Related Works
Prior work has advanced vision-language, audio-text, and omni-modal models, but existing systems still struggle to integrate modalities while maintaining balanced performance. Ola targets these limitations through architecture, training, and data preparation.
- Vision-language models have expanded from images to video, 3D, and mixed visual inputs, but integrating audio while preserving balanced performance remains difficult.
- Audio-text research covers speech inputs and generation alongside audio understanding for music and events.
- Omni-modal models increasingly unify audio, image, video, and text, yet often lack full modality or output coverage and suffer poorer performance.
- Ola addresses these limitations with improved architecture, training strategy, and data preparation.
3. Ola: Omni-Modal Understanding
Ola is designed as a unified framework for visual, audio, textual, and video inputs. Its modality connectors, visual and audio encoders, cross-modal video data, and progressive training strategy support omni-modal learning.
- Advanced Visual Understanding: Its visual system preserves arbitrary image and frame aspect ratios and uses Local-Global Attention Pooling to reduce visual token length with less information loss.
- Architecture: Ola processes text, images, videos, and audio simultaneously through an omni-modal architecture.
- Advanced Visual Understanding: Visual pre-training combines independently conducted ViT pre-training with instruction pre-training to improve visual-language capabilities.
- Robust Audio Integration: The audio module processes speech, music, and video content through a dual-encoder design.
- Cross-Modal Alignment: Cross-modal video data captures relationships between video and audio through video-audio question answering and video speech recognition.
- Robust Audio Integration: MLP connectors project visual and audio features into unified tokens that can be freely combined with text tokens for LLM decoding.
- Progressive Alignment: Progressive training begins with image-text and expands through video frames, speech, and video with audio.
4. Experiments
Ola is evaluated across image, video, and audio understanding benchmarks, with analyses of its architecture, training, data, and cross-modal learning. Results show competitive multimodal performance, gains from audio and joint learning, and stronger responses when audio complements visual input.
- Benchmark setup: Ola is benchmarked across representative image, video, and audio understanding tasks against specialized, comprehensive, and open omni-modal language models.The evaluation includes mainstream image, video, and audio benchmarks and comparisons with modality-specific and omni-modal baselines.
- Image and video results: 84.3% on MMBench-1.1, 70.8% on MMStar, 57.0% on MMMU, 68.4% on MathVista, 86.1% on AI2D, and 827 on OCRBench demonstrate strong image understanding.These results surpass the relative multimodal language models with similar parameter counts.
- Image and video results: 68.4% on VideoMME establishes Ola as a state-of-the-art 7B model on that benchmark while handling video and audio inputs simultaneously.Ola also maintains a leading position against LLaVA-Video and VideoCCAM on LongVideoBench and MVBench.
- Audio results: 3.1% mean WER on LibriSpeech and 6.41 mean score on AIR-Bench show strong speech recognition and conversational audio ability.The audio results outperform existing omni-modal models, including LLaMA-Omni.
- Cross-modal analysis: 68.4% overall VideoMME accuracy with audio exceeds 67.1% with original text subtitles, indicating that audio can provide information beyond subtitles on some benchmarks.Adding audio to raw video increased performance from 64.4% to 68.4%, after omni-modal training increased it from 63.8% to 64.4%.
- Training analysis: Progressive image-to-video-to-audio training preserves previously learned capabilities, while joint learning improves MMBench-1.1 and VideoMME performance.The analysis supports progressive modality training and joint modality learning as effective parts of Ola’s training strategy.
- Qualitative analysis: Qualitative examples show precise speech recognition, emotional analysis, and more accurate visual responses when audio reveals speaker nationality, context, and background.Vision-language models without audio inputs exhibit information loss in the interview example.
5. Conclusion
The paper concludes that Ola provides a competitive open pipeline for omni-modal understanding across image, video, and audio tasks. Its architecture, curated data, joint learning, and progressive modality alignment connect the supported modalities.
- Conclusion: Ola achieves competitive performance across image, video, and audio understanding tasks.The conclusion presents this as the paper’s central outcome.
- Conclusion: Architectural design, data curation, and training strategy form a natural, efficient, and competitive pipeline for building an omni-modal model.The approach combines omni-modal inputs, high-quality pre-training and fine-tuning data, joint modality learning, and progressive modality alignment.
- Conclusion: The authors hope Ola will inspire future research on more general AI models.
A.1. Model Details
Ola combines a Qwen-2.5-7B language model with dedicated visual and audio encoders and connector modules for projecting modality features into the language-model space.
- Encoders: The visual encoder uses SigLIP-400M features fine-tuned for native-resolution visual inputs, with patch size 16 and hidden dimension 1152.The encoder contains 27 transformer blocks and 16 attention heads.
- Encoders: The audio encoder combines Whisper-V3-Large and BEATs for audio processing.The model uses these components for speech and music-related inputs.
- Connector layer: Two separate 2-layer MLP connectors project visual and audio features into the language model’s dimensionality.Each connector’s input matches its encoder dimension and its output matches the LLM dimension.
- Connector layer: Local-Global Attention Pooling uses a predictor over concatenated features, with a dimension reduced from 2× to 1×.
- Language model: Qwen-2.5-7B supplies the language model, with hidden dimension 3,584, intermediate size 18,944, and 28 transformer layers.
A.2. Training Details
Ola uses progressive modality alignment in three stages, moving from image-text training to image-video training and then joint audio integration. Each stage introduces additional modalities while retaining the staged alignment procedure.
- Stage 1: Image-text: Stage 1 trains image-text adapters using 808k LAION image-caption examples before supervised fine-tuning.Connector parameters are unfrozen during adapter pre-training while other parameters remain frozen; the learning rate is 1e-3 and batch size is 256.
- Stage 2: Image-video: Stage 2 adds video to image-text supervised fine-tuning with 2.7 million total examples, including 1.9M videos and 800k image-text pairs.The learning rate is 2e-5, maximum sequence length is 16k, and maximum frames are 64.
- Stage 3: Audio integration: Stage 3 initializes a speech adapter through projector alignment and pre-trains it on 370k LibriTTS examples.Speech-adapter parameters are unfrozen while other parameters remain frozen, using learning rate 1e-3 and batch size 256.
- Stage 3: Audio integration: After speech-adapter pre-training, Stage 3 combines image, video, and pure audio data for audio-video joint alignment.
A.3. Data Collection Details
Ola’s data collection combines large-scale image-text, video-audio, and multimodal instruction data to support cross-modal understanding. The pipeline filters video subtitles and generates audio-relevant question-answer pairs for omni-modal alignment.
- ViT pre-training: 10M multimodal samples support ViT pre-training across OCR-intensive, image-text, and video-text domains.The visual-text alignment uses Qwen2.5-0.5B as the language interface and cross-entropy loss over joint embedding representations.
- Instruction pre-training: 20 million image-text pairs form the instruction-pretraining dataset, combining open-source, proprietary, and knowledge-intensive sources.GPT-4o and Gemini-Pro optimize question-answer pairs for greater complexity and multi-hop reasoning.
- Video-audio data: Qwen-2-72B generates at least three video-grounded questions and answers per video using subtitles and additional visual context.The prompt requires relevance to the subtitle and video and prohibits fabricated content.
- Omni-modal alignment: 243k cross-modal video-audio data points from 81k videos are used for stage 3 omni-modal alignment.The collected examples support training that connects visual, audio, and textual information.
B. More Analysis
Cross-modal video-audio training improves video performance while leaving image understanding largely stable. Both academic and open-ended video sources provide gains over an audio-only stage-3 baseline.
- Cross-modal training data: 65.7% VideoMME accuracy results from adding open-ended video-audio data, compared with 59.0% without video training.Academic video-audio data reaches 64.2% on the same benchmark.
- Cross-modal training data: Video-audio training minimally affects image benchmarks, indicating stable image understanding after text-image training.The ablation compares a baseline without video training against academic and open-ended video sources.
- Cross-modal training data: LibriSpeech ASR performance improves with video-audio data, which the authors associate with challenging subtitling tasks in complex environments.The reported explanation is framed as a likely contribution of the cross-modal data.
C.1. Text and Audio Understanding
Ola demonstrates cross-modal understanding on mixed audio-text and video inputs. Its video capability jointly uses video, audio, and text to gather information from videos.
- Text and Audio Understanding: Ola performs strongly on music-, speech-, and sound-related inputs in mixed audio-text understanding.The examples are presented as practical visualizations of cross-modal audio and text instruction following.
- Video Understanding: Jointly recognizing video, audio, and text allows Ola to gather more information from video inputs.The paper presents these results alongside comparisons with state-of-the-art vision language models.
D. Limitations
Ola supports omni-modal understanding through text generation but does not explore audio generation. This limits applicability in scenarios where audio generation is crucial.
- Limitations: Ola focuses solely on text generation and does not explore audio generation for omni-modal understanding.The authors identify audio generation as a direction for future work.
- Limitations: The lack of audio generation may restrict Ola’s applicability where audio generation is crucial.This limitation concerns the model’s ability to fully leverage multimodal inputs.