Ola: Pushing the Frontiers of Omni-Modal Language Model

• Ola’s innovations target visual and audio understanding within an omni-modal language model.Source: Figure 1 · p. 1
• The paper acknowledges equal contribution among marked authors.Source: §1 · p. 1
• Figure 2 compares Ola with open-source multimodal, omni-modal, and GPT-4o models across image, video, and audio benchmarks.Source: Figure 2 · p. 2

• Existing open omni-modal models retain a substantial performance gap from specialized LLMs, limiting their progress toward real-world applications.Source: §1 · p. 2
• Ola combines an extensible omni-modal architecture, visual and audio encoders, joint alignment, and curated multimodal data.Source: §1 · p. 2
• Video serves as the central bridge because visual content, audio, and subtitles are highly correlated within samples.Source: §1 · p. 2
• With 7B parameters, Ola achieves 68.4% accuracy on VideoMME using video and audio inputs.Source: §1 · pp. 2–3
• Ola reports 72.6% mean accuracy on OpenCompass, 3.1 mean WER on LibriSpeech, and 6.41 GPT-eval score on AIR-Bench.Source: §1 · p. 3

• Vision-language models have expanded from images to video, 3D, and mixed visual inputs, but integrating audio while preserving balanced performance remains difficult.Source: §2 · p. 3
• Audio-text research covers speech inputs and generation alongside audio understanding for music and events.Source: §2 · p. 3
• Omni-modal models increasingly unify audio, image, video, and text, yet often lack full modality or output coverage and suffer poorer performance.Source: §2 · p. 3
• Ola addresses these limitations with improved architecture, training strategy, and data preparation.Source: §2 · p. 3

Advanced Visual Understanding

• Its visual system preserves arbitrary image and frame aspect ratios and uses Local-Global Attention Pooling to reduce visual token length with less information loss.Source: §3.1 · p. 3

Architecture

• Ola processes text, images, videos, and audio simultaneously through an omni-modal architecture.Source: §3 · p. 3, Figure 3 · p. 4

Advanced Visual Understanding

• Visual pre-training combines independently conducted ViT pre-training with instruction pre-training to improve visual-language capabilities.Source: §3.1 · p. 4

Robust Audio Integration

• The audio module processes speech, music, and video content through a dual-encoder design.Source: §3.2 · pp. 4–5

Cross-Modal Alignment

• Cross-modal video data captures relationships between video and audio through video-audio question answering and video speech recognition.Source: Figure 4 · p. 5, §3.3.1 · pp. 5–6

Robust Audio Integration

• MLP connectors project visual and audio features into unified tokens that can be freely combined with text tokens for LLM decoding.Source: §3.2 · p. 5

Progressive Alignment

• Progressive training begins with image-text and expands through video frames, speech, and video with audio.Source: §3.3 · p. 5, §3.3.2 · p. 6

Benchmark setup

• Ola is benchmarked across representative image, video, and audio understanding tasks against specialized, comprehensive, and open omni-modal language models.The evaluation includes mainstream image, video, and audio benchmarks and comparisons with modality-specific and omni-modal baselines.Source: §4 · p. 6, §4.2 · p. 7

Image and video results

• 84.3% on MMBench-1.1, 70.8% on MMStar, 57.0% on MMMU, 68.4% on MathVista, 86.1% on AI2D, and 827 on OCRBench demonstrate strong image understanding.These results surpass the relative multimodal language models with similar parameter counts.Source: §4.2 · p. 7
• 68.4% on VideoMME establishes Ola as a state-of-the-art 7B model on that benchmark while handling video and audio inputs simultaneously.Ola also maintains a leading position against LLaVA-Video and VideoCCAM on LongVideoBench and MVBench.Source: §4.2 · pp. 7–8

Audio results

• 3.1% mean WER on LibriSpeech and 6.41 mean score on AIR-Bench show strong speech recognition and conversational audio ability.The audio results outperform existing omni-modal models, including LLaMA-Omni.Source: §4.2 · p. 8

Cross-modal analysis

• 68.4% overall VideoMME accuracy with audio exceeds 67.1% with original text subtitles, indicating that audio can provide information beyond subtitles on some benchmarks.Adding audio to raw video increased performance from 64.4% to 68.4%, after omni-modal training increased it from 63.8% to 64.4%.Source: §4.3 · p. 9

Training analysis

• Progressive image-to-video-to-audio training preserves previously learned capabilities, while joint learning improves MMBench-1.1 and VideoMME performance.The analysis supports progressive modality training and joint modality learning as effective parts of Ola’s training strategy.Source: §4.3 · p. 9

Qualitative analysis

• Qualitative examples show precise speech recognition, emotional analysis, and more accurate visual responses when audio reveals speaker nationality, context, and background.Vision-language models without audio inputs exhibit information loss in the interview example.Source: §4.3 · p. 9

Conclusion

• Ola achieves competitive performance across image, video, and audio understanding tasks.The conclusion presents this as the paper’s central outcome.Source: §5 · p. 9
• Architectural design, data curation, and training strategy form a natural, efficient, and competitive pipeline for building an omni-modal model.The approach combines omni-modal inputs, high-quality pre-training and fine-tuning data, joint modality learning, and progressive modality alignment.Source: §5 · p. 9
• The authors hope Ola will inspire future research on more general AI models.Source: §5 · p. 9

Encoders

• The visual encoder uses SigLIP-400M features fine-tuned for native-resolution visual inputs, with patch size 16 and hidden dimension 1152.The encoder contains 27 transformer blocks and 16 attention heads.Source: §A.1 · p. 10
• The audio encoder combines Whisper-V3-Large and BEATs for audio processing.The model uses these components for speech and music-related inputs.Source: §A.1 · p. 10

Connector layer

• Two separate 2-layer MLP connectors project visual and audio features into the language model’s dimensionality.Each connector’s input matches its encoder dimension and its output matches the LLM dimension.Source: §A.1 · p. 10
• Local-Global Attention Pooling uses a predictor over concatenated features, with a dimension reduced from 2× to 1×.Source: §A.1 · p. 10

Language model

• Qwen-2.5-7B supplies the language model, with hidden dimension 3,584, intermediate size 18,944, and 28 transformer layers.Source: §A.1 · p. 10

Stage 1: Image-text

• Stage 1 trains image-text adapters using 808k LAION image-caption examples before supervised fine-tuning.Connector parameters are unfrozen during adapter pre-training while other parameters remain frozen; the learning rate is 1e-3 and batch size is 256.Source: §A.2 · p. 10

Stage 2: Image-video

• Stage 2 adds video to image-text supervised fine-tuning with 2.7 million total examples, including 1.9M videos and 800k image-text pairs.The learning rate is 2e-5, maximum sequence length is 16k, and maximum frames are 64.Source: §A.2 · p. 10

Stage 3: Audio integration

• Stage 3 initializes a speech adapter through projector alignment and pre-trains it on 370k LibriTTS examples.Speech-adapter parameters are unfrozen while other parameters remain frozen, using learning rate 1e-3 and batch size 256.Source: §A.2 · p. 10
• After speech-adapter pre-training, Stage 3 combines image, video, and pure audio data for audio-video joint alignment.Source: §A.2 · p. 10

ViT pre-training

• 10M multimodal samples support ViT pre-training across OCR-intensive, image-text, and video-text domains.The visual-text alignment uses Qwen2.5-0.5B as the language interface and cross-entropy loss over joint embedding representations.Source: §A.3 · p. 10

Instruction pre-training

• 20 million image-text pairs form the instruction-pretraining dataset, combining open-source, proprietary, and knowledge-intensive sources.GPT-4o and Gemini-Pro optimize question-answer pairs for greater complexity and multi-hop reasoning.Source: §A.3 · p. 10

Video-audio data

• Qwen-2-72B generates at least three video-grounded questions and answers per video using subtitles and additional visual context.The prompt requires relevance to the subtitle and video and prohibits fabricated content.Source: §A.3 · p. 11

Omni-modal alignment

• 243k cross-modal video-audio data points from 81k videos are used for stage 3 omni-modal alignment.The collected examples support training that connects visual, audio, and textual information.Source: §A.3 · pp. 11–12

Cross-modal training data

• 65.7% VideoMME accuracy results from adding open-ended video-audio data, compared with 59.0% without video training.Academic video-audio data reaches 64.2% on the same benchmark.Source: B. More Analysis · p. 12
• Video-audio training minimally affects image benchmarks, indicating stable image understanding after text-image training.The ablation compares a baseline without video training against academic and open-ended video sources.Source: B. More Analysis · p. 12
• LibriSpeech ASR performance improves with video-audio data, which the authors associate with challenging subtitling tasks in complex environments.The reported explanation is framed as a likely contribution of the cross-modal data.Source: B. More Analysis · p. 12

Text and Audio Understanding

• Ola performs strongly on music-, speech-, and sound-related inputs in mixed audio-text understanding.The examples are presented as practical visualizations of cross-modal audio and text instruction following.Source: §C.1 · p. 12

Video Understanding

• Jointly recognizing video, audio, and text allows Ola to gather more information from video inputs.The paper presents these results alongside comparisons with state-of-the-art vision language models.Source: §C.2 · p. 13

Limitations

• Ola focuses solely on text generation and does not explore audio generation for omni-modal understanding.The authors identify audio generation as a direction for future work.Source: D. Limitations · p. 13
• The lack of audio generation may restrict Ola’s applicability where audio generation is crucial.This limitation concerns the model’s ability to fully leverage multimodal inputs.Source: D. Limitations · p. 13