Source-linked AI summary
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, Yuming Lou, Dong Yang, Zhijian Liu, Yukang Chen, Ambrish Dantrey, Ehsan Jahangiri, Sreyan Ghosh, Daguang Xu, Ehsan Hosseini-Asl, Danial Mohseni Taheri, Vidya Murali, Sifei Liu, Yao Lu, Oluwatobi Olabiyi, Yu-Chiang Frank Wang, Rafael Valle, Bryan Catanzaro, Andrew Tao, Song Han, Jan Kautz, Hongxu Yin, Pavlo Molchanov
TL;DR
OmniVinci addresses the challenge of building an effective omni-modal LLM that jointly understands vision, audio, and language. It combines shared-space alignment, temporal modeling, and a 24M-sample data pipeline, achieving gains across cross-modal, audio, and vision benchmarks with 0.2T training tokens. The paper also reports benefits for perception, reasoning, and downstream applications.
Problem
Training an omni-modal system that jointly understands vision, audio, and language remains expensive and challenging because it requires suitable architecture and data choices.
Method
OmniVinci combines OmniAlignNet, temporal embedding techniques, and a curation-and-synthesis pipeline producing 24 million multimodal conversations.
Results
+19.05% on Dailyomni, +1.7% on MMAR, and +3.9% on Video-MME are reported over prior art, using 0.2T training tokens.
Takeaways & Limitations
The findings report that audio and video reinforce one another in perception and reasoning, with benefits extending to robotics, medical, broadcasting, and smart-factory applications.
Takeaways & Limitations
OmniAlignNet aligns high-level visual and audio semantics but does not by itself model their temporal relationships.
Abstract
from arXiv · showhide
Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.
1. Introduction
OmniVinci systematically develops an omni-modal LLM that jointly understands vision, audio, and language through architectural and data-design choices. It reports broad benchmark gains over prior systems while using substantially fewer training tokens.
- 1. Introduction: The study examines architecture, data curation, and training strategy choices for simultaneous understanding of vision, audio, and language.
- 1. Introduction: The architecture aligns vision and audio in a shared omni-modal space using OmniAlignNet, Temporal Embedding Grouping, and Constrained Rotary Time Embedding.These techniques target semantic alignment, timestamp-based grouping, and periodic temporal information.
- 1. Introduction: The data pipeline curates 24 million multimodal conversations spanning audio, video, and image tasks, including modal-specific and omni-modal samples.It combines existing video-with-audio question-answering data with synthetic conversations carrying explicit omni-modal labels.
- 1. Introduction: +19.05% on Dailyomni, +1.7% on MMAR, and +3.9% on Video-MME are reported gains over prior art for joint, audio, and vision understanding.The paper also reports +2.83% on WorldSense.
- 1. Introduction: OmniVinci uses 0.2T training tokens, around 6× fewer than Qwen2.5-Omni’s 1.2T tokens.
2. Model Architecture
OmniVinci integrates vision and audio into a shared latent space, aligning their semantics and temporal relationships before LLM processing. Its architecture combines OmniAlignNet, Temporal Embedding Grouping, and Constrained Rotary Time Embedding to represent complementary cross-modal information and both relative and absolute time.
- Omni-Modal Alignment Mechanism: OmniAlignNet maps visual and audio sequences into a shared latent space, then uses bidirectional contrastive learning to align their representations.Query embeddings, self-attention, and L2 normalization produce fixed-size modality-shared embeddings before symmetric vision-to-audio and audio-to-vision losses are applied.
- Temporal Embedding Grouping: Temporal Embedding Grouping orders visual and audio embeddings by timestamp so the LLM can capture relative temporal relationships across modalities.The grouped sequence is formed by combining modality-specific temporal groups in chronological order, and experiments report improved multimodal comprehension.
- Constrained Rotary Time Embedding: Constrained Rotary Time Embedding encodes absolute timestamps with a maximum time horizon, balancing sensitivity to fine-grained differences against broader temporal shifts.CRTE constructs base frequencies, modulates them by timestamps, and applies element-wise rotary embedding; smaller horizons emphasize local differences, while larger ones capture broader trends.
- Constrained Rotary Time Embedding: The architecture passes temporally aligned omni-modal embeddings to the LLM, enabling integration of fine- and coarse-grained timing cues.Different embedding dimensions rotate at different frequencies, supporting multi-scale temporal representations while preserving the original embeddings’ semantic structure.
- Input-Output Configuration: The model supports flexible combinations of video, audio, speech, text, and image inputs, with text outputs optionally connected to text-to-speech systems.The design targets applications such as spoken video descriptions, spoken-question answering, and verbal robot instruction.
3. Training Strategy
OmniVinci uses staged training that first develops modality-specific capabilities, then integrates vision and audio through joint training. Its data strategy combines existing modality-specific and implicit omni-modal data with explicitly synthesized cross-modal supervision and corrected captions.
- A two-stage strategy first trains vision and audio comprehension independently, then performs omni-modal joint training to integrate those capabilities.
- The joint-training mixture combines modality-specific conversations with omni-modal data containing both visual and audio inputs.
- Existing video-QA datasets provide implicit omni-modal supervision because videos naturally contain simultaneous visual and audio streams.
- An omni-modal data engine synthesizes explicit labels for videos with audio tracks, providing more direct supervision for joint visual-audio understanding.
- Joint captioning corrects modality-specific hallucinations by combining independently generated visual and audio captions into comprehensive summaries across clips.
- 24 million conversations span image, video, and audio tasks, with omni-modal data contributing 15% and image data the largest share at 36%.
4. Experiments
Experiments validate OmniVinci’s alignment and joint visual-audio learning designs, then show strong omni-modal, audio, video, visual, reasoning, and downstream-task performance.
- 4.1.1. Visual-Audio Alignment Scheme: +2.34 average improvement from 50.25 to 52.59 shows OmniAlignNet adds gains after TEG and CRTE.The full alignment design also improves Omnibench by +6.1, Worldsense by +0.75, and Dailyomni by +1.17.
- 4.1.2. Implicit and Explicit Learning: Explicit omni-modal data added after implicit audio-video learning yields stronger improvements across benchmarks.Implicit learning uses 270K video conversations with audio, while explicit learning comes from the omni-modal data engine.
- 4.2.1. Omni-Modal Benchmark: +4.07 average-score improvement over Qwen2.5-Omni on omni-modal benchmarks, with +19.05 on Dailyomni and +2.83 on Worldsense.OmniVinci reaches 53.73 average, 66.50 on Dailyomni, and 48.23 on Worldsense.
- Audio QA: +1.7 on MMAR and +0.6 on MMAU over Qwen2.5-Omni demonstrate improved general audio understanding.The model also achieves WERs of 1.7 on LibriSpeech-clean and 3.7 on LibriSpeech-other.
- Video Understanding: +3.1 on Video-MME over Qwen2.5-VL-7B accompanies competitive video results, including 70.6 on MVBench.The Video-MME score is 68.2 without subtitle hints.
- Omni-Modal Reasoning: Joint audio-visual input surpasses visual-alone input for GRPO training, producing faster and better convergence.Both models benefit from multimodal RL, while OmniVinci surpasses Qwen2.5-Omni on the GRPO accuracy curve within 15 steps.
- 4.4. Downstream Tasks: OmniVinci improves applications involving video-audio perception, including robot navigation, medical analysis, and semiconductor-factory monitoring.The study also reports sports video understanding and cross-lingual speech translation applications.
5. Conclusion
OmniVinci is an open-source omni-modal LLM designed for joint perception across images, videos, audio, and text. The paper reports frontier multimodal performance, lower costs, and downstream applications including speech-driven navigation.
- OmniVinci addresses the challenge of building a unified omni-modal model through architectural innovations and enhanced data and training recipes.
- OmniVinci supports joint perception across images, videos, audio, and text while improving downstream agentic applications.
- Robotics: Speech-Driven Vision Language Navigation: Speech-driven navigation is motivated by the practicality of commanding robots through natural human speech rather than text-only prompts.
- Robotics: Speech-Driven Vision Language Navigation: The model targets speech-driven navigation, where spoken instructions are paired with visual observations and predicted movement actions.On R2R-CE, the speech-driven model achieves comparable performance to text-driven NVILA with lower navigation error.
B.2. Sport Video Understanding
OmniVinci is evaluated across sports, speech translation, clinical video understanding, and semiconductor manufacturing. The reported applications combine multimodal perception with temporal reasoning, synchronized audio-visual interpretation, and domain-specific analysis.
- Sport Video Understanding: The model demonstrates strong sports-video understanding despite its compact 9-billion-parameter scale and generates detailed tennis event captions.
- Tennis Broadcasting: OmniVinci outperforms Qwen2.5-Omni on tennis point outcomes and rally length in high-resolution broadcast clips.The tennis benchmark uses clips from 14 full matches, with multiple-choice and open-ended questions spanning 3–120 seconds.
- Speech Translation: On CoVoST2, OmniVinci delivers competitive speech translation quality, including 23.2 BLEU for Japanese X→EN and 23.0 BLEU for Arabic X→EN.
- Medical AI: OmniVinci gains about +2.0 percentage points over Qwen2.5-Omni across four medical video categories.The largest margin is +6.1 in temporal reasoning, with +0.7 gains in long-horizon reasoning, audio-visual synchronization, and anti-shortcutting.
- Semiconductor Manufacturing: For wafer defect analysis, OmniVinci surpasses VILA and NVILA and is positioned for interactive querying and root-cause reasoning.
- Smart Factory: OmniVinci reaches 87% accuracy on held-out time-series fault classification after transforming signals into visual representations.The model recognizes spikes, drops, persistent runs, trends, cyclic oscillations, mixtures, random fluctuations, and missing values or outages.
C.2.1. Vision Training
Vision training progressively aligns visual inputs with the language model, expands visual encoding and multimodal pre-training, and finishes with image and video instruction tuning. This produces a vision preliminary checkpoint with trained vision and language components.
- Stage 1 | Vision Projector Alignment: Vision projector alignment makes visual embeddings compatible with language-model token embeddings using image-text captioning supervision.Only the vision projector is tuned in this first stage.
- Stage 2 | Vision Encoder Alignment: Vision encoder alignment then trains the vision encoder and projector to process diverse visual content.
- Stage 3 | Vision Pre-Training: Vision pre-training fine-tunes the projector and LLM on large-scale multimodal data while keeping the vision encoder frozen.
- Stage 4 | Image Instruction Tuning: Image instruction tuning teaches multimodal question answering, captioning, scene reasoning, document interpretation, and related visual capabilities.
- Stage 5 | Video Instruction Tuning: The five-stage vision pipeline ends with video instruction tuning for temporal reasoning and sequence-level visual understanding.The final stage covers activity recognition, multi-frame object tracking, and time-sensitive questions while fine-tuning the whole model.
- The alignment process yields a vision preliminary checkpoint containing a trained vision encoder, projector, and language model.
C.2.2. Audio Training
Audio training begins from the vision preliminary checkpoint and adds audio alignment followed by end-to-end audio instruction tuning. The authors report that audio-focused training can reduce visual-task ability, motivating subsequent omni-modal joint training.
- Audio training first aligns the audio encoder and projector, then fine-tunes the audio encoder, projector, and language model jointly.
- Stage 1 | Audio Projector & Encoder Alignment: Audio projector and encoder alignment keeps the language and vision sides fixed while learning unified representations from heterogeneous audio-language data.The 50K training pairs cover music, non-speech sound, speech question answering, speech-to-text captioning, and automatic speech recognition.
- Stage 2 | Audio Instruction Tuning: Audio instruction tuning uses a 9.6-million-sample audio-supervised dataset and jointly optimizes the audio components with the language model.
- Visual understanding becomes worse after audio instruction tuning, motivating subsequent omni-modal joint training.
C.3. Omni-Modal Joint Training Details
Training uses a cosine learning-rate schedule with a 3% linear warm-up, while freezing the vision and audio encoders. The total training budget is approximately 200 billion tokens.
- A cosine learning-rate schedule follows a linear warm-up over the first 3% of training.
- 200 billion tokens comprise the approximate total training budget.
- The vision and audio encoders remain frozen during training.
C.4. Extra Details of Training Data
The training corpus covers speech, sound, image, video, and textual tasks through approximately 24 million multimodal samples. Its distribution includes both modality-specific and omni-modal conversations.
- The corpus covers speech recognition, translation, emotion classification, audio question answering, audio captioning, video question answering, and image understanding.
- The mixture includes 3.6 million omni-modal, 8 million image-text, 2.7 million video-text, and 5.3 million speech-text conversations.
- Omni-modal data contributes 15% of the training mixture, including omni question answering and omni captioning.
- Approximately 24 million samples span Speech, Sound, and Image/Video categories.
D.1. Audio Encoding
Audio encoding uses AF-Whisper and investigates token compression to balance speech and audio understanding with computational efficiency. Downsampling reduces latency and improves long-audio accuracy, while deployment optimizations accelerate inference.
- AF-Whisper consistently outperforms Qwen-2 Audio on audio and speech understanding tasks, motivating its use in the final architecture.
- A 30-second audio produces 750 feature vectors, or roughly 25 tokens per second, making one hour require about 90,000 tokens.
- 1.7× lower time-to-first-token latency and 2.72× faster decoding let the 8B model process up to 64 video frames on a 24GB RTX 4090 GPU.
- Halving audio tokens reduces latency from 1.78 sec/sample to 1.40 sec/sample, a 17.7% improvement.
- Audio downsampling improves long-audio accuracy by 2% while compressing information into more condensed embeddings.
D.3. OmniVinci with ASR Test-Time Scaling Methods
OmniVinci extends speech recognition at test time with cascaded ASR and retrieval-augmented correction. These methods let the model choose between its internal hypothesis and external transcription evidence, improving WER in the reported experiments.
- Results: The cascaded pipeline adds improvements on ASR tasks and is particularly beneficial for offline transcription.
- Results: OmniVinci-RAG reduces average WER from 6.3 to 5.0, outperforming the cascaded pipeline’s reduction to 5.7 at the same ASR model size.
- Method: The system combines audio, an internal hypothesis, and external candidate transcriptions to generate corrected text.
- Method: A control token selects whether to accept the internal hypothesis or integrate external references before generating the final transcript.
- Method: Training assigns the decision label by comparing internal and external hypotheses with ground-truth transcripts and their WER.
- Inference-Time Control Flow: At inference, <accept_internal> uses the internal hypothesis, whereas <integrate_reference> triggers generation of a corrected transcript.
D.4. Speech Output
OmniVinci’s speech-output evaluation compares prompting strategies and positions, while also assessing speech naturalness, intelligibility, and deployment considerations. Native audio prompting and suffix placement are favored for robust prompt following, while synthesis back ends mainly affect speech surface realization.
- 3.32 MOS and 8.2% WER mark Bark’s weaker speech-rendering performance on both naturalness and intelligibility measures.MOS is higher-is-better, whereas WER is lower-is-better.
- The evaluation reports prompt adherence, constrained-command slot accuracy, latency, MOS, and WER across interleaved audio–vision prompting setups.The compared strategies are transcript, native audio, and TTS-injected prompting, with prefix, mid, and suffix positions also ablated.
- Native audio prompting is most robust to accents, background noise, and overlapped speech, yielding higher prompt adherence and slot accuracy in difficult conditions.Transcript prompting remains competitive on clean speech but degrades when ASR struggles with named entities or code-switched fragments.
- Suffix-position audio prompts consistently outperform prefix and mid insertion, likely because they reduce long-range interference in the attention context.
- Encoding audio directly without external ASR provides the best prompt following under realistic noise and accents while reducing latency and memory.Suffix-position prompts provide the strongest control.
- Agentic cascaded synthesis tends to combine strong MOS with low WER, while autoregressive systems remain competitive but more variable across regimes.Changing the TTS back end affects speech surface realization without altering language understanding or response planning.