Source-linked AI summary
OneLLM: One Framework to Align All Modalities with Language
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, Xiangyu Yue
TL;DR
Existing multimodal LLMs rely on modality-specific encoders that are difficult to unify and typically cover only common modalities. OneLLM uses a shared encoder, routed universal projection module, progressive alignment, and multimodal instruction tuning to support eight modalities, with strong results across diverse evaluations. Its main limitations are weaker data coverage beyond images and insufficient fine-grained understanding for high-resolution or long-sequence inputs.
Problem
Existing multimodal LLMs rely on architecturally different modality-specific encoders and are generally limited to common modalities.
Method
OneLLM shares a universal encoder and universal projection module across eight modalities, progressively aligns them to language, and finetunes on a multimodal instruction dataset.
Results
OneLLM achieves excellent performance across 25 diverse multimodal captioning, question answering, and reasoning benchmarks, including outperforming specialized and multimodal baselines in reported tasks.
Takeaways & Limitations
A unified encoder and routed projection interface can support one model spanning eight modalities while retaining multimodal understanding, reasoning, and instruction-following capabilities.
Takeaways & Limitations
OneLLM lacks large-scale, high-quality datasets for modalities beyond images and has limited fine-grained understanding of high-resolution images and long audio or video sequences.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have gained significant attention due to their strong multimodal understanding capability. However, existing works rely heavily on modality-specific encoders, which usually differ in architecture and are limited to common modalities. In this paper, we present OneLLM, an MLLM that aligns eight modalities to language using a unified framework. We achieve this through a unified multimodal encoder and a progressive multimodal alignment pipeline. In detail, we first train an image projection module to connect a vision encoder with LLM. Then, we build a universal projection module (UPM) by mixing multiple image projection modules and dynamic routing. Finally, we progressively align more modalities to LLM with the UPM. To fully leverage the potential of OneLLM in following instructions, we also curated a comprehensive multimodal instruction dataset, including 2M items from image, audio, video, point cloud, depth/normal map, IMU and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks, encompassing tasks such as multimodal captioning, question answering and reasoning, where it delivers excellent performance. Code, data, model and online demo are available at https://github.com/csuhan/OneLLM
1. Introduction
OneLLM addresses the difficulty of extending multimodal LLMs beyond common modalities by sharing a universal encoder and projection module across eight modalities. It combines progressive alignment with modality-specific tokenizers and a large multimodal instruction dataset.
- Existing multimodal LLMs commonly connect separate modality-specific encoders and projection modules to the language model.
- Pretrained transformers may serve as universal cross-modal encoders, reducing the need for separately pretrained encoders for every modality.
- OneLLM shares an encoder and projection module across modalities while lightweight convolutional tokenizers convert each input signal into tokens.
- OneLLM progressively aligns additional modalities by first training an image-based vision LLM, then mixing image projection experts with dynamic routing.
- The curated instruction dataset covers captioning, question answering, and reasoning across image, audio, video, point cloud, depth/normal map, IMU, and fMRI.
2. Related Work
Related work extends language models to vision and other modalities, but most multimodal systems retain modality-specific encoders and alignment modules. OneLLM instead follows a unified-encoder direction and targets eight modalities.
- Large Vision-Language Models: Several vision-language models project visual tokens directly or resample them into fixed-length sequences to reduce language-model computation.
- Large Vision-Language Models: OneLLM uses learnable modality tokens to aggregate inputs and generate fixed-length tokens for all modalities.
- Multimodal Large Language Models: Prior multimodal systems commonly connect separate pretrained image, audio, and video encoders to LLMs through modality-specific Q-Formers, Perceivers, or adapters.
- Multimodal-Text Alignment: The paper positions a universal encoder and projection module as an approach for mapping multimodal inputs into the LLM.
3. Method
OneLLM uses modality-specific tokenizers with a frozen universal encoder and a routed projection interface, followed by progressive alignment and unified instruction tuning. The LLM is fully finetuned only during instruction tuning.
- Model Architecture: Each modality tokenizer uses a 2D or 1D convolution layer to convert an input signal into a token sequence for the transformer encoder.
- Model Architecture: The universal projection module combines multiple projection experts, while a soft router dynamically weights their contributions for each input.
- Model Architecture: Learnable modality tokens select the modality and summarize variable-length inputs into fixed-length projected tokens supplied to the LLM.
- Progressive Multimodal Alignment: Progressive alignment begins with image-text pretraining and then grounds video, audio, point cloud, depth/normal map, IMU, and fMRI modalities in stages.
- Unified Multimodal Instruction Tuning: The instruction-tuning dataset combines modality-specific captioning, question answering, and reasoning data for unified multimodal finetuning.
- Unified Multimodal Instruction Tuning: During instruction tuning, OneLLM fully finetunes the LLM while keeping the remaining parameters frozen.
4. Experiment
OneLLM is evaluated across diverse multimodal tasks and modalities, with results indicating strong performance from a unified model. Ablations further examine joint training, image initialization, projection experts, and router design.
- Video-Text Evaluation: OneLLM outperforms both multimodal and video-specific models on video question answering tasks.The training data excludes video QA datasets with answer options, although it includes similar visual question-answering data.
- Audio-Text Evaluation: OneLLM outperforms ChatBridge and LTU on Clotho Caption, while its zero-shot Clotho AQA result is on-par with fully finetuned Pengi.The paper distinguishes dataset-specific captioning performance from zero-shot question-answering performance.
- Depth/Normal-Map Evaluation: OneLLM achieves superior zero-shot scene-classification accuracy to CLIP on depth/normal-map data and adapts from synthetic training data to real-world scenarios.The evaluation uses NYUv2 and SUN RGB-D; both provide depth maps, with normal maps generated using a pretrained DPT model.
- Separate Training vs. Joint Training: Jointly trained audio and video models perform much better than separately trained models on ClothoQA and MSVDQA, respectively.The authors attribute this benefit to transferring learned question-answering knowledge across modalities, particularly for data-scarce modalities.
- Alignment and Projection Design: Image-text pretraining improves multimodal alignment, three projection experts support all modalities, and soft routing outperforms constant and sparse routers.Directly aligning all modalities with a randomly initialized universal projection module substantially reduces image and video performance.
5. Conclusion
OneLLM aligns eight modalities with language through a unified framework, progressive alignment, and a large-scale multimodal instruction dataset. It is evaluated broadly, while remaining limited by data scarcity beyond images and fine-grained understanding of complex inputs.
- OneLLM aligns eight modalities with language using a universal encoder, universal projection module, LLM, and progressive alignment pipeline.The framework is designed to handle multimodal inputs with a single model.
- The curated multimodal instruction dataset supports captioning, question answering, and reasoning across the covered modalities.
- OneLLM is evaluated on 25 diverse benchmarks, showing excellent performance.
- Limitation and Future Work: The absence of large-scale, high-quality datasets beyond images creates a performance gap with specialized models on those modalities.
- Limitation and Future Work: Fine-grained multimodal understanding remains challenging for high-resolution images and long video or audio sequences.
B. Additional Ablation Experiments
The ablations examine universal-encoder choices and trainability. Frozen CLIP-ViT is preferred because trainable CLIP harms visual performance and increases memory use, while DINOv2 underperforms due to lacking language alignment.
- Frozen CLIP-ViT is the default universal encoder, while the ablations compare trainable CLIP-ViT and DINOv2.
- Frozen vs. Trainable Encoder: Trainable CLIP reduces image and video performance significantly but improves ClothoQA audio QA by 4.7%.
- Frozen vs. Trainable Encoder: Frozen CLIP uses 46Gb of memory versus 74Gb for trainable CLIP, making the frozen setting preferable for this framework.
- Beyond Vision-Language Encoder: DINOv2 produces lower OneLLM performance than CLIP-ViT because it is not aligned with language.Using DINOv2 requires learning vision-language alignment from scratch.
C. Additional Implementation Details
OneLLM uses lightweight modality-specific tokenizers to convert heterogeneous inputs into token sequences compatible with a shared transformer encoder. The tokenizers use convolutional transformations tailored to each modality.
- The modality tokenizer transforms each input signal into a sequence of tokens for the transformer encoder.
- Visual Tokenizer: Visual inputs use one shared 2D convolution tokenizer for images, videos, and depth/normal maps, with video frames processed in parallel.Images and depth/normal maps are treated as one-frame video inputs.
- Audio Tokenizer: Audio is converted into a 2D spectrogram with default shape 1×128×1024 before applying a single 2D convolution tokenizer.
- Point Tokenizer: Point clouds are sampled to 8192 points, grouped into 512 groups of 32 points, then encoded by convolution and max pooling.The resulting output tokens have shape R1024×1024.
- IMU and fMRI Tokenizers: IMU inputs use a 1D convolution with Cin=6, Cout=1024, K=10, and S=1, while fMRI inputs use a 1D convolution mapping 15724 channels to 8196.The fMRI output is resized to R1024×8 for transformer input.
C.2. Multimodal-Text Alignment Dataset
The multimodal-text alignment data combines modality-specific sources and annotations for depth/normal maps, IMU, and fMRI. These datasets support aligning heterogeneous signals with language.
- Depth/normal-text pairs use roughly 0.5M CC3M image-text pairs, with DPT generating the depth/normal maps.
- IMU-text pairs combine Ego4D sensor data with corresponding video narrations as text annotations.
- fMRI-text pairs use the NSD subj01 imaging session, with MS COCO image captions annotating the visual stimuli.
C.3. Multimodal Instruction Tuning Dataset
This section summarizes the multimodal instruction-tuning dataset and points to the associated training prompt formats.
- C.3. Multimodal Instruction Tuning Dataset: The multimodal instruction-tuning dataset is summarized in Table 9.
- Dataset summaries and dataset-specific prompt formats are documented in separate tables.Table 9 summarizes the dataset, while Table 10 presents the training prompts.
- C.4. Prompt Design: The training prompt formats for each dataset are shown in Table 10.
D. Evaluation Details
The evaluation section specifies dataset prompts and protocols across image, video, audio, point cloud, depth/normal-map, IMU, and fMRI tasks, alongside multimodal LLM comparisons.
- Evaluation prompts for each dataset and additional evaluation details are introduced in Table 11.
- Image, Video and Audio Tasks: QA tasks predict option letters or open-ended words and phrases, while captioning tasks generate one-sentence captions.
- Image, Video and Audio Tasks: For audio-video-text tasks, the LLM input sequence is video tokens, audio tokens, then text prompts.
- Point Cloud Tasks: Point-cloud classification follows PointLLM’s prompt and evaluates accuracy using GPT4.
- Depth/Normal Map Tasks: Depth/normal-map scene classification prepends a category list and asks OneLLM to choose one class.
- IMU/fMRI Tasks: IMU and fMRI evaluations use captioning prompts asking for motion descriptions or scene descriptions based on fMRI data.
F. Additional Qualitative Results
This section provides additional qualitative results in Figures 4, 5, and 6.
- Additional qualitative results are presented in Figure 4.
- Additional qualitative results are presented in Figure 5.
- Additional qualitative results are presented in Figure 6.