Source-linked AI summary
Ovis: Structural Embedding Alignment for Multimodal Large Language Model
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Han-Jia Ye
TL;DR
MLLMs commonly connect pre-trained vision and language models even though their embedding strategies differ. Ovis aligns them by mapping visual patches to probabilistic tokens that index a learnable visual embedding table, and it outperforms comparable open-source models while surpassing Qwen-VL-Plus overall. The paper also identifies limitations for high-resolution and multi-image visual understanding.
Problem
MLLMs use structured textual embeddings from a lookup table but unstructured continuous visual embeddings, leaving their tokenization and embedding strategies misaligned.
Method
Ovis maps each visual patch to a probabilistic token that indexes a learnable visual embedding table multiple times, producing a weighted combination aligned with textual embeddings.
Results
Ovis outperforms open-source MLLMs at similar parameter scales across various benchmarks, and Ovis-14B surpasses Qwen-VL-Plus overall.
Takeaways & Limitations
The results support structured alignment of visual and textual embeddings as a promising direction for MLLM architectural design.
Takeaways & Limitations
Ovis has limited efficacy on high-resolution visual tasks and is trained only on single-image samples, challenging multi-image understanding.
Abstract
from arXiv · showhide
Current Multimodal Large Language Models (MLLMs) typically integrate a pre-trained LLM with another pre-trained vision transformer through a connector, such as an MLP, endowing the LLM with visual capabilities. However, the misalignment between two embedding strategies in MLLMs -- the structural textual embeddings based on an embedding look-up table and the continuous embeddings generated directly by the vision encoder -- makes challenges for a more seamless fusion of visual and textual information. We propose Ovis, a novel MLLM architecture designed to structurally align visual and textual embeddings. Ovis integrates an additional learnable visual embedding table into the visual encoder's process. To capture rich visual semantics, each image patch indexes the visual embedding table multiple times, resulting in a final visual embedding that is a probabilistic combination of the indexed embeddings. This structural approach mirrors the method used for generating textual embeddings. Empirical evaluations on various multimodal benchmarks show that Ovis outperforms open-source MLLMs of similar parameter scales and even surpasses the proprietary model Qwen-VL-Plus overall. These results highlight the potential of Ovis' structured visual representation for advancing MLLM architectural design and promoting more effective multimodal learning. Code, datasets, and models are available at https://github.com/AIDC-AI/Ovis.
1 Introduction
MLLMs commonly connect pre-trained vision and language models despite differing embedding strategies. Ovis introduces structured visual embeddings that mirror textual embeddings and reports strong benchmark performance.
- MLLMs combine language comprehension with visual perception to extend LLMs beyond text.
- Connector-based MLLMs align visual and textual embedding dimensions, but retain different tokenization and embedding strategies across modalities.
- Ovis introduces a learnable visual embedding table that transforms continuous visual tokens into structured embeddings aligned with textual embeddings.
- Each visual patch becomes a probabilistic token that indexes the visual embedding table multiple times, producing an expectation-weighted combination of embeddings.
- Ovis jointly uses textual generation loss and three-stage optimization instead of an additional image autoencoder with vector quantization and other losses.
- Ovis outperforms same-tier open-source MLLMs across most evaluated benchmarks, while Ovis-14B surpasses Qwen-VL-Plus overall.
2 Related Work
Related work situates Ovis among LLM-based multimodal architectures and visual-tokenization methods. The paper emphasizes the distinct design space of integrating structured visual tokens with language modeling.
- Multimodal Large Language Models: MLLMs extend LLMs by interpreting visual elements and relating them to textual descriptions.
- Multimodal Large Language Models: Most open-source MLLMs combine a vision encoder, a connector, and an LLM, with connectors including cross-attention-based and query-based designs.
- Visual Tokenization: Visual tokenization methods have used discrete latent variables, vector quantization, adversarial training, reconstruction losses, and decoders for visual tasks.
- Visual Tokenization: Ovis is illustrated with separate visual and textual embedding tables, probabilistic visual tokens, and weighted combinations of selected embeddings.
3 Ovis
Ovis structurally aligns visual and textual embeddings by converting visual patches into probabilistic tokens and combining entries from a learnable visual embedding table. It feeds these table-derived visual embeddings alongside textual embeddings into the LLM and trains the architecture with textual generation loss across three stages.
- 3.2 Probabilistic Visual Tokens: Ovis first transforms each visual token into a probability distribution over K visual words using a linear projection followed by softmax.The distribution represents a patch’s similarities across the visual vocabulary.
- 3.3 Visual Embedding Table: Ovis introduces a visual embedding table with one embedding vector per visual word and gives it the same dimension as the textual embedding table.This parallels the table-based generation of textual embeddings.
- 3.2 Probabilistic Visual Tokens: Each visual patch indexes multiple visual words, whose embeddings are combined by the distribution to preserve richer patch semantics than a single argmax selection.The resulting weighted combination is the final visual embedding for the patch.
- 3.3 Visual Embedding Table: The multimodal input replaces the image indicator with visual embeddings while retaining textual embeddings, so all tokens share dimensionality and a similar embedding-table generation process.The LLM then processes the combined sequence to produce textual outputs.
- 3.4 Training Strategy of Ovis: Ovis is trained in three stages with textual generation loss, progressing from visual caption training to visual descriptions and finally multimodal instruction learning.The first two stages keep the LLM frozen, while the third unfreezes and trains the entire model.
4 Experiments
Ovis is evaluated across general and specialized multimodal benchmarks against open-source and proprietary MLLMs, including controlled comparison with a conventional connector architecture. It consistently achieves strong results across parameter tiers and task types.
- Experimental Setup: Ovis uses three training stages with textual generation loss, while its configurations combine an LLM, a ViT backbone, and a visual vocabulary.The stages differ in trainable parameters and training-data types; the implementation uses open-source backbones.
- Experimental Setup: Ovis is evaluated on general benchmarks including MMMU, MMBench-EN, MMBench-CN, and MMStar, plus specialized benchmarks including MathVista-Mini, MME, HallusionBench, and RealWorldQA.The evaluation uses the VLMEvalKit package, with benchmark comparisons summarized in Tables 1 and 2.
- Main Results: Ovis-8B outperforms similarly sized open-source models across the majority of benchmarks, while Ovis-14B surpasses Qwen-VL-Plus in most benchmarks.Ovis-14B performs consistently strongly across the reported general multimodal evaluations.
- Main Results: Ovis performs strongly on specialized tasks, including mathematical and logical reasoning, real-world visual tasks, MME, and hallucination benchmarks.Ovis-14B’s RealWorldQA score exceeds GPT4V, while Ovis-8B and Ovis-14B lead their respective parameter tiers on MME and hallucination benchmarks.
- Ablation Study: With identical backbones, parameter counts, and training datasets, Ovis-7B outperforms a two-layer MLP connector-based architecture by 8.8% on average.The controlled comparison uses Qwen1.5-7B-Chat and Clip-ViT-L/14@336px backbones.
5 Conclusion
Ovis structurally aligns visual embeddings with textual embeddings by introducing a visual embedding look-up table and probabilistic visual tokens. Across multimodal benchmarks, it outperforms similarly scaled open-source MLLMs and Qwen-VL-Plus overall.
- Conclusion: Ovis maps image patches into probabilistic tokens that index an additional visual embedding table, paralleling textual embedding generation.This architecture is designed to address the different tokenization and embedding strategies used by visual and textual components.
- Conclusion: Across various multimodal benchmarks, Ovis outperforms open-source MLLMs of similar parameter scales and surpasses Qwen-VL-Plus overall.The conclusion presents these evaluations as evidence of Ovis’ effectiveness.
6 Broader Impact and Limitations
Ovis may improve visual-textual interaction, but its use can involve hallucination, bias, and other potential harms. The paper also identifies limitations in high-resolution and multi-image visual understanding.
- Broader Impact: Ovis may generate misleading or incorrect information through hallucination and may also suffer from biases and other potential harms.The paper identifies content moderation and transparent model development as possible mitigation approaches.
- Limitations: Ovis’ high-resolution visual-task efficacy is limited because it lacks high-resolution-boosted techniques.The limitation is noted despite strong RealWorldQA results with a 336px ViT backbone.
- Limitations: Because Ovis is trained only on single-image samples, it faces challenges in scenarios requiring visual understanding across multiple images.The authors plan to improve high-resolution image handling and multi-image processing in future iterations.
A Qualitative Results
Ovis-Llama3-8B performs well across varied multimodal tasks in qualitative examples drawn from the literature.
- Ovis-Llama3-8B performs well across various multimodal tasks using images and prompts sourced from literature.
C In-house Visual Description Dataset
The in-house visual description dataset contains images from diverse categories, paired with concise descriptions generated by vision-language APIs under a unified prompt.
- Images sampled from Laion and CC12M span nature, lifestyle, humanities, architecture, cartoons, and abstract art.
- Gemini-Pro or GPT-4V generated concise descriptions requiring relevant OCR while avoiding embellishment and interpretation.
D In-house Visual Instruction Dataset
The in-house visual instruction dataset uses self-questioning and answering to produce diverse, high-quality image-based questions and answers.
- Gemini-Pro and GPT-4V generated visual instruction samples from CC12M images through a self-questioning and answering process.
- Examples require multimodal reasoning, including inferring a movie’s era from attire and poster design.
- Examples also require comparing calorie counts and relating differences to shake ingredients.
E Sparsity of Ovis’ Visual Tokenizer
Ovis’ visual tokenizer produces highly sparse probability distributions over its visual vocabulary across 10,000 ImageNet-1K images.
- 0.22% of probability values exceed the 1e-4 threshold across 10,000 ImageNet-1K images.
- Each image is converted into a sequence of visual tokens, with each token represented as a probability distribution over the visual vocabulary.