PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, Sijin Wu, Ji Luo, Shen Yan, Shuai Peng, Sihang Yuan, Chaoyi Huang, Yi Lin, Yangqiu Song

arXiv:2605.13831v1cs.CV

TL;DR

Long-context LVLM training lacks sufficiently studied recipes for constructing and balancing multimodal data mixtures. This paper systematically studies LongPT, identifies effective data-design principles, and introduces MMProLong, which improves long-document VQA and generalizes beyond its 128K training window to broader multimodal long-context tasks.

  • Problem

    Practical recipes for synthesizing and balancing long-context LVLM training data remain insufficiently explored.

  • Method

    The paper systematically studies LongPT by extending Qwen2.5-VL-7B from 32K to 128K and comparing long-document VQA, OCR transcription, task mixtures, sequence lengths, and short-context data.

  • Results

    MMProLong improves long-document VQA by 7.1% at 64K and 128K, maintains strong performance at 256K and 512K without additional training, and transfers to broader multimodal long-context tasks.

  • Takeaways & Limitations

    Long-document VQA provides diverse retrieval and reasoning supervision that supports data-efficient context extension and transfer beyond the training window.

  • Takeaways & Limitations

    The study focuses primarily on 7B/8B-scale LVLMs, leaving scaling to substantially larger models and longer windows open.

Abstract

from arXiv · show

Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in agentic workflows. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; and iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.1% and maintains strong performance at 256K and 512K contexts beyond its 128K training window, without additional training. It further generalizes to webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding without task-specific supervision. Overall, our study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models.

1 Introduction

The paper studies how to train LVLMs for long contexts, addressing limited practical guidance on long-document data and mixture design. Its LongPT study finds that long-document VQA, balanced length distributions, retrieval-heavy mixtures, and instruction-formatted long data support effective and transferable context extension.

  • Long-context LVLM training recipes remain insufficiently explored, especially for synthesizing data, mixing tasks, incorporating short-context data, and choosing length distributions.
  • Long-document VQA is substantially more effective than OCR transcription for long-context continued pre-training.The VQA tasks provide instruction-formatted supervision spanning information extraction and complex numerical reasoning.
  • Balanced sequence-length data outperforms target-length-focused data near 128K, favoring retrieval across varied lengths and positions.
  • Retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity.
  • Pure long-document VQA largely preserves short-context capabilities, reducing the need for short-context mixing.
  • 7.1%: MMProLong improves long-document VQA performance at 64K and 128K contexts after 5B-token LongPT.It maintains strong performance at 256K and 512K without additional training or adaptation, exceeding baselines by over 20%.

2 Related Work

Prior work extends long-context capability through lightweight context-window methods or continued pre-training, while multimodal LongPT recipes remain underexplored. Existing evaluations span documents, needle retrieval, compression, and videos, motivating broad assessment of MMProLong.

  • Context windows are extended either with lightweight positional or attention modifications or through continued pre-training.
  • Recent LVLM reports provide limited detail on how long-context capability is built, leaving practical LongPT recipes underexplored.
  • Multimodal long-context benchmarks cover long-document VQA, needle-in-a-haystack retrieval, vision-text compression, and long-video understanding.
  • The paper evaluates MMProLong on MMLongBench, VTCBench, and long-video benchmarks to assess broad generalization.

3 Experimental Setup

The experiments use Qwen2.5-VL-7B and extend its context window from 32K to 128K through continued pre-training. Runs use a fixed 5B-token budget and Dynamic-NTK-based positional scaling.

  • Qwen2.5-VL-7B is extended from a 32K context window to 128K for the LongPT experiments.
  • The mRoPE base frequency is scaled from 1 × 10^6 to 4 × 10^6 following the Dynamic-NTK heuristic.
  • Each LongPT run uses a fixed 5B-token budget, a maximum sequence length of 131,072 tokens, and a global batch size of 4M tokens.

4 Multimodal Long-Context Data Curation

The paper curates multimodal long-context data from rendered documents using a segment-level synthesis pipeline. Short coherent segments generate QA pairs that are reintegrated into full-document contexts, with anchors added to reduce ambiguity.

  • Document Pool: The data source is a document pool of over 1.5 million PDFs spanning academic, technical, and domain-diverse documents.Documents selected for synthesis contain 32 to 50 pages.
  • Document Pool: PDF pages are rendered to images at DPI = 144 and parsed into layout-aware blocks with an OCR expert model.The parsed blocks support both long-context task categories.
  • Long-Document VQA Synthesis: The segment-level pipeline generates QA pairs from coherent 8–15-page sections and places them back into the original full-document context.The model answers from a localized segment while processing the complete document.
  • Long-Document VQA Synthesis: Sampling short segments makes QA generation efficient and avoids requiring the generator to process full documents.The paper reports that the resulting QA pairs are high quality and manually checked.
  • Long-Document VQA Synthesis: Full-document evaluation can make locally valid QA pairs ambiguous when the same information appears in multiple sections.The pipeline addresses this by adding explicit segment anchors such as section names or page ranges.

+ OCR Expert🤖

The paper constructs long-document training data by parsing documents with OCR, synthesizing segment-level VQA, and generating OCR transcription tasks. Controlled comparisons show that instruction-formatted long-document VQA provides stronger downstream supervision than OCR transcription under the same 5B-token budget.

  • Long-document VQA synthesis: The synthesis pipeline parses each document with an OCR expert, samples a coherent segment, and inserts an LVLM-generated QA pair into the original document.This forms a long-context training instance while preserving the document as the surrounding context.
  • Long-document VQA data: Long-document VQA includes single-page extraction, multi-page extraction, and reasoning tasks with increasing evidence complexity.The tasks require retrieval from one page, aggregation across pages, or numerical and logical operations over extracted information.
  • OCR transcription data: OCR transcription uses rendered page images as input and parsed text elements as targets, requiring long-distance image-text dependency modeling.The data include full-document OCR and needle-page OCR, where only 1–3 pages contain the transcription targets and the rest act as distractors.
  • Comparison: The evaluation compares VQA and OCR data on MMLongBench's document category at 64K and 128K across MMLongBench-Doc, LongDocURL, and SlideVQA.The table also distinguishes models with and without an extra 5B-token SFT stage.
  • Comparison: Under a controlled 5B-token budget, long-document VQA consistently improves downstream performance by more than 5% absolute, while full-document OCR average performance drops by 17.4% to 33.17%.Multi-page extraction achieves the best average of 56.90%; OCR-trained checkpoints require an additional 5B-token SFT stage for moderate gains.

5 Data Mixture and Training Design

The study evaluates LongPT design choices for sequence lengths, task mixtures, and short-context preservation. Balanced length distributions and retrieval-heavy mixtures perform best, while pure long-context VQA preserves short-context ability with only a mild average drop.

  • Training Sequence-Length Distribution: Pool-native length distributions outperform long-biased distributions despite containing fewer near-128K examples.Average gains are +1.3, +0.1, and +1.7 points for extract-single, extract-multi, and reasoning tasks, respectively.
  • Training Sequence-Length Distribution: LongPT should train key-information retrieval across diverse lengths and positions rather than specialize to a single target length.The findings suggest continuous calibration across absolute positions and relative image-text distances.
  • Multi-Task Long-Context Data Mixture: An extraction-to-reasoning ratio of 8:2 achieves the best long-context mixture performance under the fixed 5B-token budget.Moderately extraction-heavy mixtures perform best, and the 8:2 mixture outperforms the best single-data setting.
  • Multi-Task Long-Context Data Mixture: Retrieving key information remains the primary bottleneck, while a modest amount of reasoning data preserves task diversity.The final mixture combines complementary extraction and reasoning tasks rather than relying on a single task.
  • Short-Context Performance Preservation: With 0% short-context data, long-document VQA averages 57.70 while short-context performance declines mildly from 66.47 to 65.48.The authors attribute this preservation possibly to the instruction-following format of long-document VQA.
  • Short-Context Performance Preservation: Adding 20% short-context data maximizes short-context performance at 66.53 but lowers long-document VQA to 55.57, whereas 40% offers a stronger balance.At 40% short-context data, long-document VQA averages 57.01 and short-context performance 66.14; the final recipe uses pure long-context training.

6 MMProLong Performance and Generalization

MMProLong achieves strong long-document VQA performance and generalizes beyond its 128K training window to longer contexts and diverse multimodal long-context tasks.

  • 6.1 MMProLong Compared with LVLM Baselines: MMProLong achieves the best overall average among open-source LVLMs below 15B parameters, improving the Qwen2.5-VL-7B base from 50.59 to 57.70.The improvement is consistent at both 64K and 128K contexts.
  • 6.2 Generalization Beyond the Training Setting: MMProLong maintains strong long-document VQA performance at 256K and 512K without additional training, raising the overall average over Qwen2.5-VL-7B from 28.80 to 53.80.It scores 55.09 at 256K and 52.52 at 512K.
  • 6.2 Generalization Beyond the Training Setting: MMProLong improves MM-NIAH’s average score from 20.0 to 49.4, with especially pronounced gains in retrieval and reasoning.MM-NIAH evaluates retrieval, counting, and reasoning over webpage-based multimodal haystacks.
  • 6.2 Generalization Beyond the Training Setting: MMProLong consistently improves over Qwen2.5-VL-7B on Video-MME, MLVU, and LongVideoBench despite using no video-specific training data.The recipe also transfers to long-context vision-text compression on VTCBench.
  • 6.2 Generalization Beyond the Training Setting: These results indicate that LongPT learns a general long-context multimodal capability rather than overfitting to document VQA.The observed transfer spans webpage retrieval, vision-text compression, and long-video understanding.

7 Conclusion

The paper studies how to construct and mix multimodal long-context data for LVLM continued pre-training, culminating in a practical recipe that improves long-document VQA and transfers across longer contexts and tasks.

  • The study systematically examines how to construct and mix effective multimodal long-context data for LVLM continued pre-training.
  • MMProLong improves long-document VQA and generalizes beyond the training window to 256K and 512K contexts and broader multimodal tasks such as long-video understanding.
  • The authors present the recipe as a practical foundation for building future LVLMs with reliable long-context capability.
  • The final recipe uses long-document VQA as the primary data source, samples sequences across 32K–128K, and applies an 8 : 2 extraction-to-reasoning mixture.

A.2 Training Implementation Details

The implementation extends Qwen2.5-VL-7B from 32K to 128K and uses multimodal pre-training infrastructure, distributed parallelism, and standardized evaluation protocols.

  • Training uses VeOmni with AdamW, Dynamic-NTK mRoPE base frequency adjustment, FlashAttention, Ulysses sequence parallelism, and FSDP.The 128K configuration uses Ulysses size 2 and FSDP size 4 across 64 H20 GPUs.
  • Evaluation uses MMLongBench v1.1 for long-document VQA and MM-NIAH, while VLMEvalKit covers VTCBench, long-video, and short-context benchmarks.
  • The main document evaluation uses MMLongBench-Doc, LongDocURL, and SlideVQA at standardized lengths from 8K through 128K.Unless otherwise specified, models are evaluated at 64K and 128K.
  • Long-document VQA scores use LLM-judged binary or list-answer evaluation, with per-length and overall averages computed across the three datasets and two context lengths.

B.2 Details of Long-Document VQA Evaluation with the Final Recipe

The evaluation compares MMProLong with diverse LVLM baselines across long-document VQA, longer-context extrapolation, multimodal retrieval, compression, video, and short-context capabilities.

  • The baseline comparison includes open-source and closed-source LVLMs spanning multiple model families and parameter scales.
  • Long-document VQA is evaluated at 256K and 512K by padding examples with randomly sampled negative documents, without additional training or inference-time adaptation.
  • MM-NIAH evaluates retrieval, counting, and reasoning over webpage haystacks across text-needle and image-needle variants.
  • VTCBench measures preservation of visual-text information under compressed visual context through retrieval, reasoning, and memory tasks.
  • Long-video evaluation uses 1fps videos with up to 768 sampled frames and reports benchmark-specific accuracy measures.
  • Short-context degradation is monitored across general VQA, multimodal reasoning, and text recognition benchmarks.
  • Data synthesis: The data-synthesis corpus comprises rendered PDF documents whose layout-aware OCR blocks provide structure signals for VQA sampling and targets for OCR baselines.
  • Data synthesis: Long-document VQA training samples cover 32–50 rendered PDF pages and approximately 32K–128K multimodal tokens.

D.2 Data Statistics for Long-Biased Distribution

The long-biased distribution samples longer documents and concentrates training examples near the upper end of the target context window. Its data is synthesized from coherent multi-page spans into several document-grounded VQA tasks, with manual checks indicating high but imperfect quality.

  • Distribution: Long-biased data samples 50–100 rendered PDF pages and places more training mass near the upper end of the target context window.The distribution is contrasted with the pool-native default.
  • Distribution: 83.9% of long-biased samples contain at least 100K tokens, compared with about 23.5%–23.6% for pool-native samples.
  • Data synthesis: Each source document contributes a semantically coherent span of 8–15 consecutive pages for document-grounded QA synthesis.OCR block labels identify title and section boundaries before sampling.
  • Data synthesis: The VQA synthesis pipeline uses a shared prompt with two in-context exemplars and task-specific descriptions for single-page extraction, multi-page extraction, and reasoning.
  • Quality control: 97 of 100 manually inspected QA pairs are fully correct, while two contain incorrect answers and one has inaccurate evidence annotation.The inspection suggests high-quality supervision with a small amount of noise.
  • Data design: The study compares long-document VQA and OCR transcription as contrasting long-context supervision families, alongside experiments varying short-context data.

G.2 Short-Data Mixing for Long-Document VQA

The short-data mixing experiment varies the proportion of short-context instruction data under a fixed 5B-token budget and reports results across long-document VQA datasets. The supplied passages also situate this experiment within broader generalization and evaluation-scope analyses.

  • Evaluation: The short-data mixing ablation reports full per-dataset long-document VQA results alongside an AVG-only summary in Figure 3.
  • Short-data mixing: Short-context data is mixed at proportions from 0% to 80% during LongPT under a fixed 5B-token budget.The long-context component uses an 8:2 extraction-to-reasoning mixture, and 0% means only long-context data.
  • Generalization: The broader LongPT study evaluates generalization to multimodal needle retrieval, vision-text compression, and long-video understanding.
  • Backbone transfer: The recipe is also tested on Qwen3-VL-8B, but that experiment is diagnostic because the backbone already has native 256K-context training and additional SFT and RL optimization.
  • Scope and limitations: The systematic study is primarily conducted at 7B/8B scale, leaving broader scaling to substantially larger models and longer context windows for future work.
  • Scope and limitations: Long-document VQA evaluation relies on model-based judging, whose API cost limits the frequency and scale of evaluation.
Loading 2605.13831v1…