Source-linked AI summary
Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, Tyler Poon, Max Ehrlich, Tuomas Rintamaki, Tyler Poon, Tong Lu, Limin Wang, Bryan Catanzaro, Jan Kautz, Andrew Tao, Zhiding Yu, Guilin Liu
TL;DR
Long-context VLMs remain challenged by extended video and high-resolution visual inputs, with existing methods limited by performance, scaling, and computational constraints. Eagle 2.5 addresses these gaps through information-preserving sampling, progressive mixed post-training, and a dual-level annotated video dataset. It achieves state-of-the-art performance across long-video and high-resolution image benchmarks, including 72.4 on VideoMME and results comparable to larger frontier models.
Problem
Long-context VLMs face unresolved challenges in long video and high-resolution understanding, including limited performance scaling and computational or capacity constraints.
Method
Eagle 2.5 uses Image Area Preservation, Automatic Degradation Sampling, progressive mixed post-training, and Eagle-Video-110K’s story-level and clip-level annotations.
Results
Eagle 2.5 achieves state-of-the-art performance across long-video and high-resolution image benchmarks, including 72.4 on Video-MME and results comparable to larger frontier models.
Takeaways & Limitations
Eagle 2.5 provides a strong foundation for long-context multimodal understanding, especially video understanding, while using fewer parameters than larger frontier models.
Takeaways & Limitations
Compression and selection alternatives can introduce computational overhead or capacity limitations, motivating Eagle 2.5’s native long-context approach.
Abstract
from arXiv · showhide
We introduce Eagle 2.5, a family of frontier vision-language models (VLMs) for long-context multimodal learning. Our work addresses the challenges in long video comprehension and high-resolution image understanding, introducing a generalist framework for both tasks. The proposed training framework incorporates Automatic Degrade Sampling and Image Area Preservation, two techniques that preserve contextual integrity and visual details. The framework also includes numerous efficiency optimizations in the pipeline for long-context data training. Finally, we propose Eagle-Video-110K, a novel dataset that integrates both story-level and clip-level annotations, facilitating long-video understanding. Eagle 2.5 demonstrates substantial improvements on long-context multimodal benchmarks, providing a robust solution to the limitations of existing VLMs. Notably, our best model Eagle 2.5-8B achieves 72.4% on Video-MME with 512 input frames, matching the results of top-tier commercial model such as GPT-4o and large-scale open-source models like Qwen2.5-VL-72B and InternVL2.5-78B.
1. Introduction
Eagle 2.5 targets the underdeveloped problem of native long-context visual understanding in long videos and high-resolution media. Its training framework combines information-preserving sampling, progressive post-training, and dual-level video data to improve performance as visual context grows.
- Long-context VLMs remain limited by dataset, architecture, training, and computation challenges across extended videos and high-resolution media.
- Existing approaches often add computational overhead or capacity limitations, while native long-context methods can underperform proprietary models and scale inconsistently with more visual input.
- Eagle 2.5 gains performance as input length increases and achieves competitive results against larger models including GPT-4o and Qwen2.5-VL-72B.
- Eagle 2.5 combines information-first sampling and progressive training to preserve multimodal information while expanding context length during post-training.
- Image Area Preservation retains most original image area with aspect-ratio fidelity, while Automatic Degradation Sampling retains complete text and adapts visual content to maximize context utilization.
- Eagle-Video-110K combines story-level chapters and dense narrative captions with clip-level question-answer pairs anchored by temporal and contextual references.
2. Related Work
Related work spans general VLM advances, context-compression or context-extension methods, and datasets for long-context multimodal understanding. Eagle 2.5 focuses on native long-context capabilities while addressing performance scaling, training, and data-recipe gaps.
- Modern open-source VLMs have advanced visual understanding and can match or exceed commercial models, but most still focus on short-context inputs.
- Long-context methods either use specialized compression or selection modules or directly extend LLM context length for longer multimodal sequences.
- Existing long-context approaches often underperform proprietary models, lack consistent gains with increasing visual input, and leave training strategies and data recipes underexplored.
- Long-context datasets cover documents, automated annotations, and hierarchical video annotation, balancing narrative preservation, multimodal assessment, and creation cost.
3. Eagle 2.5
Eagle 2.5 is a general multimodal system for efficiently processing long-context inputs. Its training framework combines information-first sampling, progressive post-training, and diverse data—including Eagle-Video-110K—to preserve textual and visual information while improving context utilization.
- 3. Eagle 2.5: Eagle 2.5 is designed as a versatile multimodal system for efficiently processing long-context information rather than as a specialized long-context model.The architecture avoids tailored compression modules to preserve adaptability and generalization across tasks.
- Information-First Sampling: Image Area Preservation optimizes tiling to retain at least 60% of the original image area while aligning the tiled and original aspect ratios.The objective penalizes configurations below the area threshold and favors aspect-ratio fidelity.
- Information-First Sampling: Automatic Degradation Sampling dynamically adjusts temporal sampling and image tiling to maximize visual information within the available context budget.It first determines temporal sampling and then selects the highest feasible tile configuration; samples below the minimum frame requirement are discarded.
- Training Strategy: Information-first sampling preserves complete textual input while allocating the remaining context budget to visual detail and temporal coverage.The strategy optimizes image tile counts and temporal sampling counts under a fixed visual token budget.
- Training Strategy: The post-training framework combines mixed and progressive mixed post-training to improve performance across varying context lengths.Mixed training uses length-balanced packing, while progressive training addresses scenarios with large maximum context lengths.
- Data Recipe: Eagle-Video-110K complements open-source data with diverse long videos annotated through both story-level and clip-level approaches.The dataset uses a diversity-driven collection strategy and combines comprehensive narrative annotations with fine-grained clip annotations.
4. Experiments
Eagle 2.5 performs strongly across video and image benchmarks, while ablations show that cross-modal data, information-first sampling, progressive training, and Eagle-Video-110K each affect long-context performance.
- Video benchmarks: Eagle2.5-8B achieves 74.8 on MVBench, 82.0 on Perception_test, 72.2 on EgoSchema, 77.6 on MLVU, and 66.4 on LongVideobench.These results surpass comparable InternVL2.5 and Qwen2.5-VL models on the reported benchmarks.
- Video benchmarks: 72.4 on VideoMME (w/o subtitle) places Eagle 2.5 above same-sized models and close to a 72B-parameter model.The model also reports strong results on CG-Bench, HourVideo, and Charade-STA.
- Image benchmarks: Eagle2.5-8B achieves 94.1 on DocVQA, 87.5 on ChartQA, 80.4 on InfoVQA, 83.7 on TextVQA, and 869 on OCRBench.The reported results cover document understanding, chart interpretation, information extraction, and optical character recognition.
- Cross-modal data: Increasing long-context data does not harm short-context image benchmarks and slightly benefits them, while image pre-training improves MVBench and MLVU more than Video-MME.Video-MME gains are less pronounced on the more challenging held-out benchmark.
- Information-first sampling: Removing Image Area Preservation degrades InfoVQA and Perception-test, while omitting Automatic Degradation Sampling risks truncating supervision signals and reducing performance.The effects of Image Area Preservation are strongest for high-resolution image and fine-grained video tasks.
- Post-training schedules: Progressive mixed training from 32K to 64K outperforms direct 64K mixed training, and its capacity to process more frames increases as training advances.The reported explanation is that direct 64K training dilutes shorter-context samples and makes longer samples harder to learn without gradual training.
- Eagle-Video-110K: Eagle-Video-110K significantly improves handling of at least 128 frames by adding long videos absent from the Open-Data training set.The dataset also enhances several mainstream long- and short-video benchmarks.
5. Conclusion
Eagle 2.5 is presented as a family of vision-language models for long-context multimodal understanding. Its training strategies and dual-level dataset annotations improve long-video and high-resolution image understanding, with video performance comparable to larger frontier models.
- Contributions: Eagle 2.5 combines information-first sampling, progressive mixed post-training, and Eagle-Video-110K with dual-level annotations.The conclusion identifies these components as supporting improvements in long-context understanding, especially for video.
- Results: Eagle 2.5 achieves state-of-the-art performance across benchmarks covering long video comprehension and high-resolution image understanding.The conclusion reports comparable video understanding to GPT-4o and Gemini 1.5 Pro despite fewer parameters.
- Implications: The work positions Eagle 2.5 as a foundation for efficient and versatile VLMs in complex real-world scenarios.
B. Training and Inference
The long-context training pipeline uses GPU-memory and distributed-context optimizations to make extended multimodal training more efficient.
- Efficiency optimizations: Eagle 2.5 optimizes long-context training with GPU memory optimization, distributed context parallelism, and video processing acceleration.
- Implementation: GPU memory optimization uses fused operators, fused linear-layer and cross-entropy computation, and CPU offloading of hidden states.Distributed context parallelism combines Ulysses and Ring communication groups.
B.2. Training Settings
Training extends Eagle-2’s Stage-1.5 model weights into a second stage for long-context learning, with progressive settings specified in the training configuration.
- Training pipeline: The pipeline reuses Eagle-2 Stage-1.5 model weights and extends long-context training in stage 2.
- Training schedule: Table 8 specifies the proposed progressive training settings.
C. Additional Benchmarks
Eagle 2.5 is additionally evaluated on SlideVQA and MMLongBench-Doc, with results reported in Tables 9 and 10.
- MMLongBench-Doc performance is reported in Table 10.
- SlideVQA performance is reported in Table 9 using the ANLS Score.ANLS denotes Approximate Normalized Levenshtein Similarity.
D. Training Data
Eagle 2.5 training uses multiple stages, progressing from MLP alignment and additional pretraining to mixtures of short- and long-context data.
- Table 10 reports performance on MMLongBench-Doc.
- Eagle2.5 training is divided into multiple stages, including Stage 1 for MLP alignment and Stage 1.5 for pretraining.Progressive long-context training is also included.
- Stage 1 uses ALLaVA, while Stage 1.5 uses the datasets summarized in Table 11.Stage 1.5 includes Eagle2.5-Image-SFT and additional pretraining data.
- Stages 2 through 4 mix short- and long-context data, including Eagle2.5-Image-SFT, Open-Data, and Eagle-Video-110K.
E.1. Data Curation Prompts
The Eagle-Video-110K curation prompts generate captions, contextual anchors, and clip- and video-level question-answer pairs from visual content and structured captions.
- E.1. Data Curation Prompts: The curation prompts cover caption generation, textual context anchors, clip-level QA, and video-level QA.
- E.1.1. Prompts for Generating Captions and Anchors: Caption prompts request brief and detailed descriptions of a video clip’s visual content.They emphasize colors, shapes, textures, people, actions, scenes, and backgrounds.
- E.1.1. Prompts for Generating Captions and Anchors: Brief captions summarize the essential action, setting, and participants in 15–30 words and treat the clip as a complete unit.The brief caption is used as a textual contextual anchor.
- E.1.1. Prompts for Generating Captions and Anchors: Detailed captions represent actions as a continuous sequence, using transitions and describing how movements connect.The requested output format contains Brief Caption and Detailed Caption fields.
- E.1.1. Prompts for Generating Captions and Anchors: Detailed captions preserve readable text in its original language with an English translation, while unclear text is mentioned without guessing its content.People are distinguished using characteristics such as clothing.
- E.1.2. Prompts for generating clip-level QA: Clip-level QA uses detailed and brief captions to generate category-specific, unambiguous question-answer pairs requiring information beyond the brief caption when possible.Categories are marked null when the answer is already available from the brief caption or neither caption supports the category.
- E.1.2. Prompts for generating clip-level QA: Clip-level QA selects question types from a defined pool and emits structured JSON with a question-answer object or null for each type.
- E.1.3. Prompts for generating generating video-level QA: Video-level QA combines clip captions with start and end times in seconds to generate category-specific, unambiguous question-answer pairs.A category is null only when the caption cannot address it; outputs use the same structured JSON format.
E.2. Question type pool
The question type pool defines categories used for QA generation, with category names and descriptions listed in Table 12.
- QA prompts generate pairs according to the category names and descriptions in the question type pool.
- Additional Stage 1.5 datasets are summarized separately.
- Table 11 summarizes datasets used for Eagle 2.5 Stage 1 and Stage 1.5, with magenta marking internal data.
- Table 12 lists the question type categories and their descriptions.