Source-linked AI summary
Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas Voegtle, Philipp Fischer, De-An Huang, Timo Roman, Tong Lu, Jose M. Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, Zhiding Yu
TL;DR
Open-source frontier VLMs often expose weights without sufficient data-strategy and implementation details for reproduction. Eagle 2 builds post-training data strategy from scratch alongside training recipes and model design, producing models that match or outperform frontier open-source and commercial VLMs on common multimodal benchmarks.
Problem
Publicly available frontier VLMs can approach commercial models, but their technical details are often insufficient for reproduction, limiting transparent development of competitive open-source systems.
Method
Eagle 2 develops a centralized post-training data strategy emphasizing diversity first, then quality, together with data refinement, training recipes, and model design.
Results
Eagle2-9B matches or outperforms frontier open-source and commercially closed-source VLMs on a set of common multimodal benchmarks.
Takeaways & Limitations
The paper provides a transparent practice for post-training data strategy intended to inspire open-source development.
Takeaways & Limitations
Some data-refinement objectives require maintaining accuracy without noticeable regression across every considered benchmark while adding domain diversity.
Abstract
from arXiv · showhide
Recently, promising progress has been made by open-source vision-language models (VLMs) in bringing their capabilities closer to those of proprietary frontier models. However, most open-source models only publish their final model weights, leaving the critical details of data strategies and implementation largely opaque. In this work, we address VLM post-training from a data-centric perspective, showing the key role of data strategy in developing frontier VLMs. By studying and building our post-training data strategy from scratch, we share detailed insights into the development processes, aiming to benefit the development of competitive models for the open-source community. Our introduced data strategy, together with training recipes and model design, leads to a family of performant VLMs named Eagle2. Specifically, Eagle2-9B achieves state-of-the-art results across various multimodal benchmarks, matching certain competitive models with up to 70B parameters.
1. Introduction
Eagle 2 argues that transparent, data-centric post-training strategies are central to building competitive open-source VLMs. It presents diverse data collection and refinement, model design, and training recipes, culminating in Eagle2 models that match or outperform frontier systems.
- 1. Introduction: Publicly available frontier VLMs can match commercial models, but their technical details are often insufficient for reproduction.
- 1.1. Data Strategy: The paper identifies post-training data strategy as the decisive factor under a fixed pretrained LLM backbone.
- 1.1. Data Strategy: Eagle 2 prioritizes diversity before quality through data collection, filtering, selection, and augmentation across 180+ sources.
- Model Architecture: The model design combines dynamic tiling with a mixture of vision encoders, using channel-concatenated encodings for high-resolution and robust perception.
- 1.3. Training Recipe: The three-stage recipe aligns modalities, trains broadly on diverse visual-instruction data, then performs high-quality visual instruction tuning.
- 1.4. Summary: Eagle2-9B matches or outperforms frontier open-source and commercial VLMs across common multimodal benchmarks despite its 9B-parameter scale.
2. Method
Eagle 2 builds VLM performance around a data-centric post-training strategy that emphasizes broad data coverage, iterative refinement, quality control, and staged training. The method combines diverse collection, similarity-based selection, filtering, augmentation, formatting, and an added Stage-1.5 pre-training stage.
- Overview: The method starts from a Cambrian-based baseline and improves it through data strategy, training recipe, and model architecture optimizations.The baseline uses 5.2M samples, an MLP connector, and image tiling for dynamic resolution.
- Data collection: Data collection prioritizes diversity through passive dataset monitoring and proactive searches guided by model error analysis.The authors also convert public non-QA data into VQA data using rules or automatic labeling.
- Data selection: The Similarity Score measures overlap between a new source and the existing pool using image and text embeddings within each data category.The score combines image and text similarity; sources below 0.3 are treated as different from the existing pool in practice.
- Data filtering: Filtering removes mismatched or irrelevant pairs, repeated texts, and numeric-formatting problems commonly found in synthesized datasets.The strategy uses rule-based filtering because many low-quality samples have distinguishable characteristics.
- Data selection: Subset selection balances source diversity and distribution by assigning smaller sampling ratios to larger, error-prone autolabeled datasets.Stage-2 averages around 20K samples per source, with VisualWebInstruct as the largest subset at 263K samples.
- Data augmentation: Augmentation enriches existing annotations with chain-of-thought explanations, rule-based question-answer generation, and expanded responses.The approach uses third-party VLMs to generate fine-grained image descriptions and other added information.
- Data formatting: Data formatting aligns similar tasks while keeping different tasks clearly distinct, because formatting choices can affect model behavior.A fixed equation template produced persistent outputs, and removing that environment significantly improved OCRBench.
3. Experiments
The experiments iteratively improve Eagle 2 through data scaling, filtering, selection, augmentation, staged training, and vision-encoder mixtures. Eagle2-9B performs competitively across 14 multimodal benchmarks, including comparisons with substantially larger and closed-source models.
- Data scaling: Data scaling improved overall performance, with the largest gains from adding 2M chart, table, and OCR VQA samples.Beyond 10M samples, experimental costs rose sharply and data-iteration efficiency declined.
- Data quality: 45 points on OCRBench followed data filtering and formatting, which also improved 8 of 14 benchmarks.The result underscores the value of reviewing existing data and cautiously using publicly available data.
- Data selection: Reducing the dataset to 4.6M balanced, high-quality samples further improved average score.Advanced selection outperformed the larger naively sampled 8.6M-data setting, whose performance declined.
- Data augmentation: Data augmentation improved MMMU and MathVista, while rule-based chart augmentation added 1 point on ChartQA.The augmentation strategy increased automatically generated chain-of-thought training data.
- Training recipe: Updating Stage-1.5 data with effective Stage-2 strategies improved ChartQA, MMVet, and MathVista.Three-stage training also made the Stage-1.5 checkpoint competitive and enabled subsequent Stage-2 training to improve the previous best model by average 3.9%.
- Model comparison: A mixture of vision encoders improved performance on 12 of 14 benchmarks, especially document, chart, and OCR tasks.Eagle2-9B outperformed InternVL2-8B and MiniCPM-v2.6 on all 14 benchmarks, led Qwen2-VL-7B on 9, and surpassed GPT-4o on ChartQA, OCRBench, and MathVista.
4. Related Work
Related work frames VLM progress around increasingly capable public-weight models, vision-centric architectures, and expanding data strategies. Eagle 2 combines tiled high-resolution input with a mixture of vision encoders while emphasizing transparent post-training data practices.
- Vision-Language Models: VLMs integrate visual features with LLMs, and public-weight models continue to make breakthroughs in visual understanding.
- Vision-Centric VLMs: Vision-centric designs emphasize strong vision foundations and high-definition input through vision encoders, mixture designs, and tiling.
- Vision-Centric VLMs: Eagle 2 introduces tiled mixture of vision encoders, combining high-resolution tiling with robust mixture-based perception and flexible vision-foundation integration.The paper describes this as the first exploration of the tiled MoVE design.
- Data Efforts in VLMs: VLM data strategies span dataset construction, balance, filtering, and training methodologies, with expanding data types and task coverage.
5. Conclusion
The paper presents a transparent, comprehensive account of post-training data strategies for frontier VLMs. It aims to provide an open practice that can inspire the community.
- The paper unveils detailed post-training data strategies for training frontier VLMs, addressing knowledge gaps around leading public-weight models.
- The authors characterize the covered data strategy as effective and comprehensive, and hope it inspires the community through transparent practice.
6. Demos
The demos illustrate Eagle2 capabilities using test cases selected directly from Qwen2-VL and InternVL2. This selection is intended to avoid cherry-picking.
- Demo cases were directly selected from Qwen2-VL and InternVL2 to avoid cherry-picking.
A. Computing
The training resource is organized around Stage-1, Stage-1.5, and Stage-2, with development effort concentrated on iterating later stages.
- Stage-1 is rarely iterated, while Stage-1.5 is usually iterated once after Stage-2 has been iterated more than 10 times.
- Stage-1, Stage-1.5, and Stage-2 are the three listed training stages.
- Table A presents the training information for Eagle2-9B.
- Table B is labeled as general VQA data.
B. Dataset
The dataset resources include detailed source counts and cover OCR, counting and grounding, science, math, and caption-and-knowledge data, with augmented data information also planned.
- The paper reports detailed sample counts for every data source across multiple dataset tables.
- The authors state that augmented data information will be provided in addition to the existing data information.
- The dataset coverage includes naive OCR data and counting-and-grounding data.
- Science, math, and caption-and-knowledge data are each represented as dataset categories.
B.1. Dataset Collection
Dataset collection combines internal multimodal resources, conversion of non-QA data, text-only datasets, and similarity-based assessment of overlap with the existing pool.
- Internal PDF OCR data contributes 500k samples to Stage-1.5 and 50k to Stage-2, while textbook data contributes 47K samples to both stages.
- Classification data is converted into multiple-choice questions, while image-only data is automatically annotated into captioning data.
- The similarity score assesses overlap between a new dataset and the existing data pool.
- The collection includes diverse open-source text-only datasets, and some preference datasets are converted into SFT format.
- UReader, COCO-Colors, and TextbookQA receive similarity scores of 0.45, 0.10, and 0.02, respectively, relative to Cambrian-7M.
- Sources scoring below 0.3 are considered different from the existing pool, while higher-scoring data may be retained or removed based on specific considerations.
B.2. Dataset Filtering
Dataset filtering uses manual inspection and rules to remove unsuitable samples, while subset selection applies explicit size thresholds and reduction guidelines.
- Filtering begins with manual inspection to identify erroneous-data features, followed by rule-based removal.
- Keyword-based rules exclude unsuitable refusals while preserving responses appropriate for safety or ethical questions.
- The paper includes a table for chain-of-thought data augmentation and reports that a step-by-step prompt can improve performance with CoT training data.
- The section references general subset-selection rules across the dataset categories listed in multiple tables.
- Datasets with fewer than 20,000 samples do not undergo subset selection.
- When subset selection is applied, at least half of the data is removed; datasets exceeding 100,000 samples are usually capped at 50,000.
B.4. Data Augmentation
Eagle2 augments existing VLM data with chain-of-thought rewriting, rule-based task expansion, answer elaboration, and balanced-aware packing. These strategies target richer reasoning, reduced hallucination, stronger spatial awareness, and less biased sequence-length distributions.
- Chain-of-Thought Augmentation: Existing Science, Math, and Chart data are rewritten by state-of-the-art VLMs to generate detailed chain-of-thought answers.The generated answers are subsequently checked against original answers by another LLM to filter erroneous samples.
- Chain-of-Thought Augmentation: Adding the prompt “Think it step-by-step” alone does not improve MathVista accuracy, whereas incorporating constructed chain-of-thought data produces a performance boost.The comparison evaluates models with and without training on the generated chain-of-thought data.
- Rule-Based Augmentation: Rule-based augmentation expands table, OCR, and other existing datasets to reduce hallucinations and enhance spatial awareness.IIIT5K and WordArt are used for OCR augmentation, while additional rules expand existing data as shown in Figure C.
- Answer Expansion: Short responses from VQAv2, GQA, and VSR are extended into detailed responses through prompting.The expansion prompt asks for more specific answers based on existing answers.
- Balanced-Aware Packing: Balanced-aware knapsack packing replaces naive greedy packing to produce more uniform knapsack length distributions.The method sorts samples, initializes redundant knapsacks, distributes samples within length limit L, and selects the currently shortest knapsack; its redundancy delta depends on data-length distribution.