Source-linked AI summary
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, Guilin Liu
TL;DR
MLLMs need stronger visual perception for complex and resolution-sensitive visual tasks, yet mixture-of-encoder designs lack systematic comparisons and ablations. This paper explores that design space, introduces pre-alignment, and finds that simple concatenation of complementary vision tokens supports Eagle’s strong benchmark performance.
Problem
Mixture-of-vision-encoder MLLMs lack systematic comparisons and detailed ablations of expert selection and multi-expert integration, despite the importance of visual perception for hallucination reduction and resolution-sensitive tasks.
Method
The paper systematically studies vision encoders, resolutions, fusion strategies, expert combinations, pre-alignment, and training recipes for MLLMs.
Results
Eagle achieves state-of-the-art performance across multiple benchmarks, with particularly strong results on OCR and document understanding tasks.
Takeaways & Limitations
Simple channel concatenation and additional complementary vision experts provide an efficient design path for improving MLLM perception, especially when encoders are unlocked.
Abstract
from arXiv · showhide
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.
1 INTRODUCTION
This study systematically explores how mixtures of vision encoders and resolutions can improve MLLM visual perception. It identifies simple, effective design choices and develops Eagle, which achieves strong benchmark performance, especially on OCR and document understanding.
- Stronger vision encoders matter for reducing hallucinations and improving resolution-sensitive tasks such as OCR, but scaling them can require substantial training resources.
- The study benchmarks encoders, compares fusion strategies, searches for complementary expert combinations, and improves expert pre-alignment and data mixtures.
- Unlocking vision encoders during MLLM training matters, contrasting with approaches that commonly freeze multiple vision encoders or teachers.
- Straightforward channel concatenation is as competitive as more advanced fusion strategies while offering better efficiency and performance.
- Adding vision experts produces consistent perception gains, particularly when the encoders are unlocked during training.
- Eagle uses pre-alignment to improve the interaction between non-text-aligned vision experts and the language model before joint training.
- Eagle attains state-of-the-art performance across multiple benchmarks, with especially clear advantages on OCR and document understanding tasks.
- Using the same pre-training and supervised fine-tuning data as Cambrian-1, Eagle models achieve better overall performance.
2 DESIGN SPACE EXPLORATION
Eagle systematically explores vision-encoder design choices for MLLMs, covering resolution adaptation, expert selection, fusion, and pre-alignment. The study identifies efficient configurations that improve multimodal perception across diverse tasks.
- Base setup: Eagle evaluates vision encoders, fusion strategies, training data, and benchmarks within a common LLaVA-1.5-based MLLM setup.The setup uses Vicuna-v1.5 7B, a vision encoder, and a projection layer, with supervised fine-tuning data collected across multiple tasks.
- Stronger CLIP encoder: 448 × 448 interpolation provides a strong efficiency-performance balance, using fewer than half the tokens of the 672 × 672 version.Updating CLIP during supervised fine-tuning improves higher-resolution performance, while slightly reducing performance at the pre-training resolution.
- Fusion paradigm exploration: Channel concatenation offers the best combination of performance, expandability, and efficiency among the compared fusion strategies.Injection-based methods are less competitive on TextVQA and OCRBench, while sequence append has comparable performance but increases sequence length substantially.
- Vision experts: Vision-language-aligned experts are more competitive than experts trained exclusively on detection, OCR, or segmentation tasks.The paper attributes this difference possibly to representational inconsistencies when vision-task experts are integrated with large language models.
- Vision-language pre-alignment: Pre-Alignment individually aligns vision experts with a frozen language model before joint training, improving performance and stabilizing training.The strategy addresses gaps between experts by aligning each representation with the same language model before combining them.
- Extension to multi-experts: Adding vision experts generally improves aggregated performance, with the selected Eagle combination comprising CLIP, ConvNeXt, SAM, Pix2Struct, and EVA-02.The paper reports that individual metrics vary, but normalized average metrics trend upward as more encoders are added.
3 EXPERIMENTS
Eagle is evaluated across visual question answering, OCR, chart understanding, and broader multimodal benchmarks using progressively refined vision-expert and training configurations. It achieves strong results, including state-of-the-art performance on GQA and VQAv2 and superior cross-category performance over Cambrian-1 with the same data.
- Eagle-X5 achieves state-of-the-art performance on GQA and VQAv2, underscoring the benefits of additional vision experts.
- The experiments compare Eagle model variants across three VQA benchmarks and benchmark OCR, document, chart, and broader multimodal capabilities.
- Eagle significantly surpasses competitors on TextVQA while supporting up to 1024x1024 resolution without complex image tiling.The reported gains are attributed to its high-resolution architecture and integration of different vision encoders.
- With high-resolution adaptation and more vision experts, Eagle identifies small text and extracts image information according to user instructions.
- The best training recipe pre-aligns each vision expert, combines them while training the projector and experts, then trains the whole model on Eagle1.8M.
- Using Cambrian-1’s pre-training and supervised fine-tuning datasets, Eagle outperforms its counterparts across all evaluated benchmarks.The strongest reported improvements are in OCR and Chart, with consistent gains in General, Knowledge, and Vision-Centric categories.
4 RELATED WORK
The related work situates Eagle among multimodal architectures, stronger vision encoders, high-resolution adaptations, and multi-encoder systems. Eagle’s related design uses multiple complementary vision experts rather than relying only on a single encoder or more elaborate fusion strategy.
- Multimodal large language models: MLLMs broadly use either cross-modal attention or prefix-tuning, and Eagle follows the LLaVA-style prefix-tuning architecture.
- Vision encoder design for MLLMs: Stronger vision encoders and high-resolution adaptations target hallucination reduction and resolution-sensitive capabilities such as OCR.
- Effectiveness of more vision experts: Figure 5 compares a CLIP+ConvNext baseline with Eagle-X5 across qualitative examples involving visual question answering and OCR-like understanding.
- Multi-encoder MLLMs: Multi-encoder models combine high-resolution features or vision models trained for complementary abilities such as text reading and object localization.
5 CONCLUSION
The conclusion presents Eagle as an in-depth study of the design space for integrating vision encoders into MLLMs. It emphasizes systematic design choices, efficient fusion, complementary vision expertise, and optimized training recipes.
- Eagle systematically studies how to integrate vision encoders for improved multimodal perception.
- The study optimizes individual encoder training recipes, identifies an extendable efficient fusion method, and combines encoders with different domain knowledge.
- The authors position the resulting design principles as a basis for future vision-encoder design in MLLMs.
A.1 BENCHMARK DETAILS
The appendix provides benchmark and ablation details for high-resolution adaptation, individual vision experts, fusion methods, encoder combinations, and training strategies. These tables document the comparisons underlying Eagle’s design choices.
- The appendix reports detailed comparisons of CLIP high-resolution adaptation methods and vision encoders across the adopted benchmarks.
- The appendix identifies the benchmark tables supporting the main comparisons, including base-data results and Cambrian-1-aligned training-data results.
- Fusion-method comparisons record visual-token counts alongside results for different vision experts.
- Training-strategy tables distinguish one-epoch supervised fine-tuning from configurations that unlock vision encoders during pre-training.
A.2 ADDITIONAL QUALITATIVE RESULTS
Eagle demonstrates qualitative reasoning on difficult VQA questions and handles OCR and document-understanding examples, including small text.
- Eagle performs reasoning based on knowledge to answer some difficult questions.
- Eagle answers difficult VQA questions requiring college-level knowledge or deliberate reasoning.
- Eagle extracts useful information from small text in OCR and document-understanding cases.