Source-linked AI summary
MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly
Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, Ginny Wong, Simon See, Pasquale Minervini, Yangqiu Song, Mark Steedman
TL;DR
Existing long-context vision-language benchmarks provide limited coverage of downstream tasks, image types, and controlled context lengths. MMLONGBENCH evaluates models across diverse tasks and standardized cross-modal lengths; its results show that single-task performance is a weak proxy for overall ability and that current models still face substantial challenges.
Problem
Existing evaluations cover a limited range of vision-language tasks and do not consistently control context length across text and image inputs.
Source: §1 · pp. 1–2, §2 · p. 3Method
MMLONGBENCH evaluates five categories of long-context vision-language tasks using unified cross-modal token counting and five standardized lengths from 8K to 128K tokens.
Source: §1 · p. 2, §3.3 · p. 5Results
Takeaways & Limitations
Abstract
from arXiv · showhide
The rapid extension of context windows in large vision-language models has given rise to long-context vision-language models (LCVLMs), which are capable of handling hundreds of images with interleaved text tokens in a single forward pass. In this work, we introduce MMLongBench, the first benchmark covering a diverse set of long-context vision-language tasks, to evaluate LCVLMs effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning five different categories of downstream tasks, such as Visual RAG and Many-Shot ICL. It also provides broad coverage of image types, including various natural and synthetic images. To assess the robustness of the models to different input lengths, all examples are delivered at five standardized input lengths (8K-128K tokens) via a cross-modal tokenization scheme that combines vision patches and text tokens. Through a thorough benchmarking of 46 closed-source and open-source LCVLMs, we provide a comprehensive analysis of the current models' vision-language long-context ability. Our results show that: i) performance on a single task is a weak proxy for overall long-context capability; ii) both closed-source and open-source models face challenges in long-context vision-language tasks, indicating substantial room for future improvement; iii) models with stronger reasoning ability tend to exhibit better long-context performance. By offering wide task coverage, various image types, and rigorous length control, MMLongBench provides the missing foundation for diagnosing and advancing the next generation of LCVLMs.
1 Introduction
Long-context vision-language models can process hundreds of images with interleaved text, but existing evaluations leave their performance across tasks and input lengths unclear. MMLONGBENCH addresses this gap with diverse tasks and standardized cross-modal context lengths, and its evaluation finds that single-task performance is a weak proxy for overall ability.
1 Introduction
- Existing benchmarks lack broad downstream-task and image-type coverage, consistent cross-modal token counting, and standardized context lengths for each example.Prior evaluations often focus on one task or image type, and some count images rather than text and image tokens together.Source: §1 · p. 2
- MMLONGBENCH covers five long-context vision-language task categories, including VRAG, many-shot ICL, and image-based document summarization.VRAG contexts use Wikipedia passages, ICL requires on-the-fly classification from hundreds of exemplars, and summarization uses image-based PDFs.Source: §1 · p. 2
- All examples use five standardized context lengths from 8K to 128K tokens, with image-token counts based on vision patches and a 2 × 2 pixel unshuffle.The benchmark also makes datasets extendable to longer contexts.Source: §1 · p. 2
- Single-task performance poorly reflects overall long-context ability; both closed- and open-source models face substantial challenges, and stronger reasoning ability tends to accompany better performance.The analysis also identifies OCR and cross-modality retrieval as bottlenecks.Source: §1 · p. 3
2 Related Work
Prior work developed methods to extend context windows in language and vision-language models, while long-context benchmarks for language models broadened beyond needle-in-a-haystack tasks. Vision-language evaluation remained narrower, motivating a benchmark that covers more long-context applications.
2 Related Work
- Researchers have extended LVLM context windows using approaches including vision-token compression, position extrapolation, and more efficient architectures.The passage also notes longer pre-training lengths as a technique for LLMs.Source: §2 · p. 3
- Long-context language-model benchmarks expanded beyond NIAH because performance on that single task only partially reflects overall long-context ability.NIAH inserts a needle into a long essay and tests recall at a specified depth.Source: §2 · p. 3
- LVLM long-context benchmarks largely focus on NIAH or long-document VQA, while MileBench contains many short-context tasks and averages about 9K tokens.The authors therefore describe existing evaluation as lacking comprehensive coverage of vision-language applications.Source: §2 · p. 3
3 Our Benchmark — MMLONGBENCH
MMLONGBENCH is designed to evaluate diverse long-context vision-language applications across image types, using unified cross-modal token counting and multiple standardized context lengths. Its datasets span five task categories and support evaluation at lengths from 8K to 128K tokens.
3 Our Benchmark — MMLONGBENCH
- MMLONGBENCH includes 13,331 examples across datasets covering key long-context capabilities and different image types.Its design criteria include diverse tasks and image types, unified token counting, and multiple standardized lengths.Source: §3 · p. 3, Table 2 · p. 4
3.1 Diverse Long-Context Applications for LCVLMs
- VRAG tests grounding in relevant retrieved information while filtering distractors, using knowledge-based VQA and Wikipedia passages to build long contexts.Wikipedia passages without the answer or named entity serve as distractors; retrieval uses the named entity as its query.Source: §3.1 · pp. 3–4
- NIAH covers image-haystack retrieval and interleaved text-image retrieval, counting, and reasoning, with accuracy or SubEM used for evaluation.For counting, the benchmark uses accuracy of summed needle counts; MM-NIAH retrieval and reasoning use SubEM and accuracy, respectively.Source: §3.1 · p. 4
- Many-shot ICL tests on-the-fly image classification across four domains, balancing exemplars per class and mapping natural-language labels to class IDs.The 128K context accommodates approximately 500 images, and each dataset uses 50 randomly sampled classes.Source: §3.1 · pp. 4–5
- Summarization evaluates concise outputs from long PDF-formatted government and legal documents using LLM-based evaluation rather than ROUGE-L.The documents are truncated from the end according to the input length.Source: §3.1 · p. 5
- DocVQA evaluates reasoning over information spread across images and text; longer documents are truncated, while shorter ones are padded with negative documents.The authors note that padding documents may occasionally contain information related to a question.Source: §3.1 · p. 5
3.2 Cross-Modality Token Counting
- The benchmark counts text tokens and image tokens together, using the Llama2 tokenizer for text and 14 × 14 image patches followed by 2 × 2 pixel unshuffle.The authors describe this method as compatible with practices in current LVLMs.Source: §3.2 · p. 5
3.3 Standardized Input Length
- Each example is provided at five standardized lengths: 8K, 16K, 32K, 64K, and 128K tokens, with extension beyond 128K possible.Lengths are controlled by adjusting passages, images, or exemplars, or by truncating PDF documents.Source: §3.3 · p. 5
4 Evaluation and Analysis
Across 46 models, MMLONGBENCH reveals that LCVLMs still struggle with long-context vision-language tasks and that performance varies across tasks. NIAH results do not consistently reflect performance across other tasks, while category correlations and error analyses highlight distinct capabilities and bottlenecks.
4.1 Evaluation on MMLONGBENCH across Tasks and Context Lengths
- At 128K tokens, GPT-4o averages 62.9, and closed-source models generally outperform open-source models, though open-source models remain competitive on some tasks.Gemini-2.5-Pro is the strongest LCVLM; its advantage over open-source models is about 20 points except on ICL, where the gap is about 14 points.Source: Figure 1 · p. 6, §4.1 · pp. 6–7
- Gemini-2.0-Flash-T improves over Gemini-2.0-Flash on all tasks, with gains of 25.3% on summarization and 10.1% on DocVQA.Gemini-2.5 models, which are natively designed as thinking models, exhibit even stronger performance.Source: §4.1 · p. 7
- Model strengths vary by task: Qwen2.5-VL-32B beats InternVL3-38B on VRAG but trails it on NIAH, while Ovis2-34B excels at summarization but struggles on DocVQA.Some models also generalize beyond their designed context windows; Qwen2-VL-72B, trained with a 32K-token window, scores 51.9 on average at 128K.Source: §4.1 · p. 7
4.2 Can Needle-in-a-Haystack Tasks Reflect LCVLM’s Overall Long-Context Ability?
- On VH-Multi, even state-of-the-art models struggle to exceed 80% accuracy at 8K tokens, and most perform only slightly above the 50% random-guess baseline.MM-NIAH counting and reasoning scores are often below 30 and 40, respectively, limiting differentiation between models.Source: Figure 3 · p. 7, §4.2 · pp. 7–8
- None of the NIAH tasks consistently correlates strongly with the broader set of long-context tasks; VH correlations are especially low, and MM-NIAH counting and reasoning correlations are weak with several downstream tasks.Simpler retrieval tasks, such as a single needle in unrelated essays, tend to correlate better across categories.Source: §4.2 · pp. 7–8
4.3 Weak Correlation Across Categories Calls for Diverse Evaluation
- At 128K tokens, categories do not consistently correlate strongly; VRAG and NIAH correlate closely, summarization and DocVQA correlate at 0.88, and ICL correlates relatively weakly with other categories.The paper relates VRAG and NIAH to their shared retrieval emphasis, and summarization and DocVQA to their shared image-based PDF input format.Source: Figure 5 · p. 8, §4.3 · p. 8
- Long-document VQA has the highest average correlation and smallest standard deviation across other categories, making it a more representative and stable proxy than NIAH.The paper notes that LongDocURL questions include complex understanding and reasoning as well as retrieval.Source: §4.3 · p. 8
4.4 Error Analysis
- DocVQA results show no clear winner between PDF and OCR-text inputs: PDFs score higher on vision-needed examples, while OCR text helps on text-pure examples, especially at 64K and 128K.In ViQuAE, replacing images with entity names improves all models; Gemma3-27B gains 26.4 points at 128K, indicating a cross-modality retrieval bottleneck.Source: Figure 6 · p. 9, §4.4 · p. 9, Figure 7 · p. 9
5 Conclusion
MMLONGBENCH provides a comprehensive benchmark for evaluating long-context vision-language models across downstream tasks. Evaluation of 46 models shows that single-task performance is not a reliable proxy for overall capability, and that challenges remain even for frontier models.
5 Conclusion
- MMLONGBENCH evaluates LCVLMs across five task categories with unified cross-modal token counting and standardized context lengths.The benchmark is presented as a foundation for diagnosing model strengths and weaknesses.Source: §5 · p. 9
- Evaluation of 46 models finds that performance on a single task does not reliably predict overall long-context capability.Source: §5 · p. 9
- Even frontier models face significant challenges, particularly in OCR accuracy and cross-modal retrieval.Source: §5 · p. 9
- Models with enhanced reasoning mechanisms, including “thinking” variants, consistently outperform their base counterparts in long-context settings.Source: §5 · p. 9
- The authors identify more efficient vision-language token encodings, robust position-extrapolation schemes, and improved multimodal retrieval and reasoning as areas for future research.Source: §5 · p. 9
NeurIPS Paper Checklist
This section records the benchmark’s checklist disclosures and describes how appendix examples are constructed across Visual RAG, MM-NIAH, and many-shot ICL. It covers reproducibility details, context-length control and scoring, and the construction of balanced examples across input lengths.
NeurIPS Paper Checklist
- The authors report no statistical-significance tests, citing the benchmark’s 20,250 evaluations per model, and say limitations are discussed in Section F.They state each model is evaluated on 4,050 examples across five input lengths; they also report that dataset construction, experiment setup, and the evaluated-model list are documented, and that data and code are open-sourced.Source: NeurIPS Paper Checklist · pp. 20–22
A Dataset Details
- The appendix provides further details on building long-context examples from existing datasets.The checklist also states that the authors report compliance with the NeurIPS Code of Ethics, include broader impact discussion, and use previously open-sourced datasets without human subjects.Source: NeurIPS Paper Checklist · pp. 23–24, A Dataset Details · p. 27
A.1 Visual Retrieval-Augmented Generation
- Visual RAG examples use Wikipedia lead sections or KILT gold passages, then control context length with hard negatives retrieved by BM25 and reranked with dense embeddings.For InfoSeek, lead sections are chunked into 100-word passages and queries lacking the answer are removed; for retrieval, question images are replaced with entity names to improve recall and produce harder distractors.Source: §A.1 · p. 27
A.2 Needle-in-a-Haystack
- Visual Haystack examples vary the number of negative distractor images to reach a target input length, counting image tokens from vision-encoder patches.The dataset includes VH-Single and VH-Multi, where target objects appear in one needle image or multiple needle images, respectively.Source: §A.2 · p. 27
- For MM-NIAH, 100-word web-page chunks enable fine-grained context-length control as text passages and images are incrementally added to reach a target length.The original contexts use full OBELICS web pages, which can contain tens of thousands of tokens; images contain only a few hundred tokens.Source: §A.2 · p. 27
- MM-NIAH evaluates retrieval, counting, and reasoning with evenly sampled text- and image-needle examples, using total-count accuracy for counting instead of exploitable soft accuracy.Retrieval and reasoning use SubEM for text needles and accuracy for image needles; the total-count measure replaces list comparison because predicting a list of zeros could score over 30 under the soft metric.Source: §A.2 · pp. 27–28
A.3 Many-Shot In-Context Learning
- Many-shot ICL uses random integer label mappings and one-per-class demonstration rounds, reducing class counts at shorter contexts to maintain similar exemplars per class.The evaluation samples 5, 10, 20, 40, and 50 classes at 8K, 16K, 32K, 64K, and 128K tokens, respectively; datasets include Stanford Cars, Food101, SUN397, and iNat 2021. iNat examples use either Birds or Plants classes. Label mapping follows established practices [20] [88], and demonstration rounds follow Li et al. [76].Source: §A.3 · p. 28
A.4 Summarization
The appendix details evaluation choices and preprocessing for summarization and long-document VQA, alongside image and model coverage. It also reports weaknesses of overlap-based evaluation and validates GPT-4o judgments against human assessments.
A.4 Summarization
- Summarization uses original PDF pages, truncates documents from the end for length control, excludes examples requiring excessive truncation, and removes GAO answer pages to prevent leakage.The two datasets contain 15,951 claims across 387 examples.Source: §A.4 · pp. 28–29
- Fluency-F1 combines precision and recall F1 with a binary fluency score, using GPT-4o to assess reference support, claim coverage, and fluency.The method decomposes gold summaries into atomic claims and evaluates generated sentences against the reference.Source: §A.4 · p. 29
- GPT-4o evaluation reveals input-length gaps and lower scores for short-context models, whereas ROUGE-L barely varies and cannot reliably distinguish summary quality.The reported ROUGE-L comparison includes InternVL3-2B and GPT-4o; the GPT-4o evaluation distinguishes performance across input lengths.Source: §A.4 · p. 29
- Human agreement with GPT-4o was 100% for fluency; precision κ was 0.90 for GovReport and 0.89 for Multi-LexSum, while recall κ was 0.90 and 0.93, respectively.Most disagreements involved partially supported cases, including inferred additions affecting precision and omitted reference details affecting recall.Source: §A.4 · pp. 29–30
A.5 Long-Document VQA
- Long-document VQA preprocessing corrects mislabeled answer formats and drops unrecoverable cases; it also excludes long-form string answers to avoid costly LLM-based extraction.After filtering, both datasets remain well-distributed.Source: §A.5 · p. 31
- VQA scoring varies by answer type, while length control preserves answer pages and pads short inputs with sampled documents that may contain question-related information.List answers are scored element by element with greedy matching; questions identify the original document to direct models to it.Source: §A.5 · pp. 31–32
A.6 Image Resizing and Statistics
- Image counts vary by task: VRAG is text-centric with one image per example, ICL is image-centric with hundreds, and MM-NIAH is intermediate.For MM-NIAH, oversized images are resized to a 1024-pixel longer edge while preserving aspect ratio.Source: §A.6 · p. 32
B Full Model List
- The model appendix describes evaluated models’ context lengths and image processing, and reports that models without pixel unshuffle cannot efficiently process long image sequences.Table 11 covers token counts for 64K and 128K ICL and Visual Haystack inputs.Source: Table 3 · p. 33, Table 10 · p. 40, Table 11 · p. 41
B.1 LVLMs Beyond our Evaluation
Some otherwise strong models were not included because of image-token inefficiency or the engineering needed to integrate their codebases.
B.1 LVLMs Beyond our Evaluation
- Models without pixel unshuffle and using dynamic tiling produce at least ten times the predefined 64K or 128K input length on high-resolution images, preventing evaluation on the authors’ GPU server.The authors could not reduce their image-token counts by resizing inputs because these models require predefined tile sizes.Source: §B.1 · p. 41
- DeepSeek-VL, DeepSeek-VL2, and Long-Llava were left for future work because their lack of a standard HuggingFace Transformers API made integration impractical within the authors’ scope.Source: §B.1 · p. 41
C Experimental Setup
The experiments evaluate proprietary and open-source models at five context lengths under the authors’ API, hardware, and sampling setup. Additional results confirm that Visual Haystack is difficult, although selected models exceed random-guess accuracy.
C Experimental Setup
- Models were evaluated at 8K, 16K, 32K, 64K, and 128K tokens; open-source evaluation used an 8×A100 server, limiting tests of models above 100B parameters at 128K.Proprietary models were evaluated through their APIs, using the versions listed by the authors.Source: C Experimental Setup · p. 41
- Sampling and positional variations yielded 4,050 examples per model across five lengths, or 20,250 evaluations per model.Single-needle tasks used six depths and multi-needle tasks used three permutations to mitigate positional bias.Source: C Experimental Setup · pp. 41–42
D.1 The Difficulty of Visual Haystack
- Gemini-2.5-Pro reaches 85.4 accuracy on VH-Single and GPT-4o reaches 70.3 on VH-Multi, both above the 50% random-guess baseline, though models perform poorly overall.The accompanying figure presents selected-model performance for both variants.Source: §D.1 · p. 42, Figure 12 · p. 42
D.2 Correlation between NIAH and Various Downstream Tasks
The additional analysis distinguishes image-needle and text-needle correlations with downstream tasks and reports weak model performance on two image-needle subsets.
D.2 Correlation between NIAH and Various Downstream Tasks
- Image-needle subsets correlate less with downstream tasks than text-needle subsets, especially for MM-NIAH-Count and MM-NIAH-Reason.Source: §D.2 · p. 42
- Models perform weakly on MM-NIAH-Count (I) and MM-NIAH-Reason (I), yielding low separability between models.Source: §D.2 · p. 42
D.3 Correlation between Datasets
Datasets generally correlate strongly within categories, with high-difficulty Visual Haystack sets and counting as exceptions. Claude evaluation excludes categories that exceed its image limit, while reasoning-tuned MiMo-VL-7B-SFT outperforms Qwen2.5-VL-7B across context lengths.
D.3 Correlation between Datasets
- Within-category dataset correlations are generally strong, but high-difficulty VH-Single and VH-Multi are exceptions, and MM-NIAH-Count correlates relatively weakly with retrieval and reasoning tasks.The authors interpret the weaker counting correlations as evidence that counting differs from retrieving or reasoning over needles.Source: §D.3 · p. 42, Figure 13 · p. 43
D.4 Performance of Claude
- Claude evaluation excludes a category at an input length when any dataset in it contains samples exceeding the 100-image request limit.The authors mark untestable cases as N/A rather than scoring them as model failures.Source: §D.4 · pp. 43–44
D.5 More Models for Reasoning Comparison
- MiMo-VL-7B-SFT (Reasoning) outperforms Qwen2.5-VL-7B across context lengths and tasks except summarization.At 8K, their average scores are 66.0 and 57.4, respectively.Source: §D.5 · p. 44
D.6 Positional Embedding Extrapolation Experiments
The tested positional extrapolation methods remain challenging for LCVLMs: YaRN helps Qwen2.5-VL-3B substantially but has little effect on larger models, while V2PE is sensitive to its visual increment and overfits NIAH. Many models also show a lost-in-the-middle pattern, with some favoring only one context endpoint at 128K.
D.6 Positional Embedding Extrapolation Experiments
- YaRN substantially improves Qwen2.5-VL-3B’s average MMLONGBENCH score at 128K, while scores for 32B and 72B fluctuate slightly.At 128K, the 3B score rises from 21.2 to 30.2; at 8K, Qwen2.5-VL-32B’s DocVQA score falls from 67.8% to 63.4%.Source: Figure 16 · p. 45, §D.6 · p. 45
- YaRN is applied zero-shot to Qwen2.5-VL, and the authors omit it from their main evaluations to preserve a fair comparison with other models.The model was pretrained at 32K, and YaRN with scaling factor 4 is used to extend the context to 128K.Source: §D.6 · p. 45
- V2PE is sensitive to visual-increment settings, and its strong NIAH performance is attributed to training on MM-NIAH, with sharp cross-category differences suggesting overfitting.Some settings perform worse than InternVL2-2B and require additional hyperparameter tuning.Source: §D.6 · pp. 45–46
D.7 Lost in the Middle
- Many LCVLMs show lost-in-the-middle behavior, and at 128K some favor the context beginning while others favor only the end.InternVL3-8B favors depth 0, whereas Qwen2.5-VL-72B favors depth 1.0.Source: §D.7 · p. 46
D.8 Error Analysis Details
The appendix specifies document-text extraction and text-pure comparison models for error analyses, and provides full model results. It also reports an Idefics2 sanity check and shows example prompts and datasets.
D.8 Error Analysis Details
- MMLongBench-Doc PDF text was extracted with PyMuPDF, while ViQuAE entity names came from the dataset; text-pure comparisons used instruction-tuned Qwen2.5 models.The text-pure models were Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct.Source: §D.8 · p. 46
D.9 Full Model Evaluation Results
- The appendix plots results for all 46 models overall and by dataset.Figures 23–27 show overall results and results for individual datasets.Source: §D.9 · p. 46
D.10 Idefics2 Performance
- Removing negative images from Visual Haystack leaves Idefics2-8B and Idefics2-8B-C performing above the 50% random-guess level.The authors describe this needle-only test as a sanity check of implementation correctness.Source: Table 13 · p. 47, §D.10 · p. 47
E Prompts and Data Examples
- The appendix provides prompt and data examples for NIAH tasks, visual retrieval, image classification, summarization, and document VQA.It includes examples from both VH and MM-NIAH because their input formats differ.Source: E Prompts and Data Examples · p. 47
F Limitation
The authors identify limits in model coverage and the cost of summarization evaluation. They describe MMLongBench as a prospective community benchmark and call for progress in token encoding, position extrapolation, OCR, retrieval, and reasoning.
F Limitation
- Some LCVLMs are omitted because of token-efficiency or codebase-integration challenges, and available compute limited open-source evaluation to models up to 72B.The authors cite difficulty evaluating models over 100B parameters at 128K with 8×A100 (80G) GPUs.Source: F Limitation · p. 47
- GPT-4o-based summarization evaluation better aligns with human judgment than ROUGE-L but is costly, limiting evaluations across hundreds or thousands of models.The authors identify a lower-cost alternative metric as needed.Source: F Limitation · p. 47
G Broader Impact
- The authors propose MMLongBench as a community evaluation standard and identify efficient vision-language token encodings, robust position extrapolation, OCR, multimodal retrieval, and reasoning as areas for model development.They connect long-context ability to applications including document understanding, web-page reasoning, long-document summarization, and dialogue grounded in visual contexts.Source: G Broader Impact · p. 47