PaperlayerView exact revision on arXiv

Source-linked AI summary

VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?

Hongbo Zhao, Meng Wang, Fei Zhu, Wenzhuo Liu, Bolin Ni, Fanhu Zeng, Gaofeng Meng, Zhaoxiang Zhang

arXiv:2512.15649v2cs.CVcs.AIcs.CL

TL;DR

Long-context methods and vision-text compression offer ways to address the computational and memory costs of expanding LLM context windows, but evidence about VLMs’ deeper understanding under compression is limited. The paper introduces VTCBench and VTCBench-Wild and finds that VLMs handle simple retrieval better than complex reasoning and long-term memory. It concludes that VTC is not a drop-in long-context solution because efficiency gains come with costs to advanced cognitive capabilities and visual robustness.

  • Problem

    Although vision-text compression can reduce input size and perform well on OCR-related tasks, its effect on VLMs’ broader long-context understanding remains under-investigated.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces VTCBench and VTCBench-Wild to evaluate VLMs on retrieval, associative reasoning, and long-term dialogue memory with compressed visual inputs.

    Source: §1 · p. 2

  • Results

    VLMs show respectable but fragile simple retrieval, while associative reasoning and long-term memory performance nearly collapse and trail text-only LLMs.

    Source: §1 · pp. 2–3, §6 · p. 15

  • Takeaways & Limitations

    VTC is not a drop-in long-context solution: its efficiency gains come at the cost of advanced cognitive capabilities and visual robustness.

    Source: §6 · p. 15

    In the thumbnail setting discussed, dense text becomes illegible after downscaling, wasting visual-processing tokens on a low-information overview.

    Source: §4.5 · p. 13

Abstract

from arXiv · show

The computational and memory overheads associated with expanding the context window of LLMs severely limit their scalability. A noteworthy solution is vision-text compression (VTC), exemplified by frameworks like DeepSeek-OCR and Glyph, which convert long texts into dense 2D visual representations, thereby achieving token compression ratios of 3x-20x. However, the impact of this high information density on the core long-context capabilities of vision-language models (VLMs) remains under-investigated. To address this gap, we introduce the first benchmark for VTC and systematically assess the performance of VLMs across three long-context understanding settings: VTC-Retrieval, which evaluates the model's ability to retrieve and aggregate information; VTC-Reasoning, which requires models to infer latent associations to locate facts with minimal lexical overlap; and VTC-Memory, which measures comprehensive question answering within long-term dialogue memory. Furthermore, we establish the VTCBench-Wild to simulate diverse input scenarios.We comprehensively evaluate leading open-source and proprietary models on our benchmarks. The results indicate that, despite being able to decode textual information (e.g., OCR) well, most VLMs exhibit a surprisingly poor long-context understanding ability with VTC-processed information, failing to capture long associations or dependencies in the context.This study provides a deep understanding of VTC and serves as a foundation for designing more efficient and scalable VLMs.

1 Introduction

Vision-text compression renders long text as compact images to reduce input tokens, but its effect on deeper long-context understanding remains an open question. The authors introduce VTCBench and VTCBench-Wild and find that VLMs handle textual perception better than complex reasoning and memory.

1 Introduction

  • VTC converts long text into dense images, achieving 3×–20× fewer input tokens than the original text sequence.The approach shifts information management from sequential attention toward spatial and visual reasoning.Source: §1 · pp. 1–2
  • Existing VTC-related benchmarks often rely on exact query-context correspondences, leaving broader long-context understanding under-investigated.The paper asks whether VLMs can understand long context after vision-text compression.Source: §1 · p. 2
  • VTCBench evaluates retrieval, associative reasoning with minimal literal overlap, and long-term dialogue memory; VTCBench-Wild adds varied real-world visual scenarios.Retrieval traces and aggregates facts; reasoning tests association beyond lexical retrieval; memory evaluates dialogue question answering.Source: §1 · p. 2
  • Most VLMs retain satisfactory OCR and simple-retrieval ability but lag text-only LLMs in long-context comprehension, especially associative reasoning and dialogue memory.The reported gap is most pronounced on tasks requiring long associations or knowledge integration.Source: §1 · pp. 2–3
  • Error analysis traces VLM failures from visual-to-textual grounding to associative reasoning, with performance critically dependent on font size and spatial layout.These sensitivities concern how information is rendered within the compressed image.Source: §1 · p. 3
  • Gemini-3-Pro outperforms some LLMs, including Qwen3-8B [61], on VTCBench-Wild and nearly matches its own text-only baseline.The authors cite this result as evidence that VTC is a viable route for long-context understanding.Source: §1 · p. 3

2 Related Work

Related work spans methods for extending or managing long contexts, visual compression of text, and benchmarks for long-context comprehension. The cited benchmark results distinguish relatively strong textual perception and simple retrieval from weaker reasoning and dialogue memory.

2.1 Long-Context Modeling

  • Long-context modeling includes architectural changes such as sparse or hierarchical attention and positional extrapolation, alongside prompt compression, retrieval, memory, and training strategies.C3 [34] uses two cascaded LLMs for text-to-latent compression rather than human-readable memory.Source: §2.1 · p. 3

2.2 Vision-Text Compression

  • Vision-text compression research renders text as images: DeepSeek-OCR [55] reports over 96% OCR precision at 9–10× compression, while Glyph [10] reports stable 3–4× compression.Other examples include VIST [59], VisInContext [48], and Li et al. [32].Source: §2.2 · p. 3
  • VTCBench and VTCBench-Wild results show stronger textual perception and simple retrieval than associative reasoning and long-term dialogue memory, where VLMs trail LLMs.The figure describes this contrast across both benchmarks.Source: Figure 2 · p. 4

2.3 Long-Context Comprehension Benchmarks

  • Long-context benchmarks range from basic Needle-In-A-Haystack retrieval to associative inference and long-term dialogue-memory evaluation.NoLiMa [47] minimizes direct lexical overlap, while other benchmarks test extended conversations.Source: §2.3 · p. 4

3 VTCBench

VTCBench evaluates long-context understanding under vision-text compression through retrieval, reasoning, and dialogue-memory tasks. Its two settings control either the compression ratio or the rendered image, separating model comparisons from a standardized visual input.

3 VTCBench

  • VTCBench includes predefined-ratio and predefined-rendering settings because different VLMs can produce different visual tokens from the same image.The benchmark covers VTC-Retrieval, VTC-Reasoning, and VTC-Memory.Source: §3 · p. 5

3.1 VTC Tasks

  • VTC-Retrieval uses dense-text images with key-value needles in long distractor-text haystacks, testing retrieval through four NIAH variants.Single-key retrieval, distractor filtering, retrieving all values for a key, and answering multiple key queries are covered.Source: §3.1 · p. 5
  • VTC-Reasoning minimizes lexical overlap between questions and needles, requiring associative links informed by real-world information or commonsense.This design tests inference beyond implicit image-to-text translation and literal matching.Source: §3.1 · p. 5
  • VTC-Memory adapts LoCoMo [36] dialogues into long multi-turn contexts with question-answer pairs, covering single-hop, multi-hop, temporal, and open-domain questions.These question types respectively concern one fact, aggregated turns, event order or timing, and world knowledge beyond the dialogue.Source: §3.1 · pp. 5–6

3.2 VTC settings

  • The predefined-ratio setting adjusts rendering for each model to hold rVTC constant, using font size to control text density and standardizing images at 896 × 896 pixels.This is intended to compare comprehension at a consistent information density while reducing preprocessing variation.Source: §3.2 · p. 6
  • The predefined-rendering setting fixes the rendering operator across models, so identical images yield model-dependent compression ratios.The preset uses 96-dpi, 12-point Helvetica, 1.2 line-height, zero margins, and 896 × 896 images; VTC-Memory also uses speaker-colored backgrounds.Source: §3.2 · p. 7

4 Experiments

The experiments compare 13 VLMs with a text-only baseline across VTC-Retrieval, VTC-Reasoning, and VTC-Memory, finding that VLMs generally struggle as long-context understanding demands increase. Performance also depends on rendering and needle position, with several distinct retrieval and reasoning failure modes.

4.1 Setup

  • The evaluation covers 13 VLMs, compares them with Qwen3-8B, and tests retrieval and reasoning at six context lengths up to 32k with needles at 11 depths.Most VLMs performed poorly at 32k, so longer texts were not evaluated.Source: §4.1 · p. 8

4.2 Main Results

  • Across tasks, VLMs generally trail text-only Qwen3-8B: retrieval weakens with length, reasoning collapses, and memory struggles on complex aggregation, with Qwen2.5-VL a notable exception.On VTC-Memory Single-Hop, Qwen2.5-VL-72B scores 55.55% ROUGE-L versus the LLM baseline’s 52.55%.Source: §4.2 · pp. 8–9

4.3 Analysis of Rendering Configuration

  • Font size is the primary rendering determinant across tasks, while font and color changes have minor effects when contrast is sufficient.Greater legibility comes at the cost of a lower compression ratio.Source: Table 2 · p. 9, §4.3 · pp. 9–10

4.4 Analysis of Needle Position

  • At 16k, VTC-Reasoning exceeds 40% accuracy at the edges but falls to 3.4% in the middle, with positional bias worsening as context length increases.The results show a U-shaped curve, with performance highest at 0% and 100% depth.Source: Figure 4 · p. 10, §4.4 · p. 10

4.5 Error Analysis

  • Reasoning errors often follow successful fact extraction; Qwen3-VL-235B-A22B-Instruct refuses more than 60% of questions in VTC-Reasoning.The authors associate these refusals with difficulty making associative inferences when the question and visual evidence lack a literal match.Source: §4.5 · p. 11, Table 3 · p. 11
  • Retrieval errors include substituting plausible distractors, and S-NIAH accuracy declines systematically as needle semantic richness diminishes.The distractor issue becomes more pronounced with longer contexts and denser distracting information.Source: §4.5 · p. 11, Table 4 · p. 11
  • On Multi-Value-NIAH, the containsAny–containsAll gap widens slightly with context length, suggesting failures to retrieve every value, not to aggregate retrieved facts.The two needles share a key and are separated by at least 20% of the context length.Source: Figure 5 · p. 12, §4.5 · p. 12
  • OCR-specific models vary with rendering and prompt: Glyph is sensitive to background color, completion prompting improves Deepseek-OCR retrieval, and dense-text thumbnails can consume tokens without useful information.InternVL3.5’s thumbnail uses 256 of 1280 tokens; the authors hypothesize that this low-information view contributes to performance degradation.Source: §4.5 · pp. 12–13

5 VTCBench-Wild

VTCBench-Wild tests whether VLMs remain robust when long-context inputs use varied visual styles, addressing the sensitivity of performance to rendering parameters. Its results show that strong text perception often does not translate into deep contextual understanding.

5.1 Construction of VTCBench-Wild

  • VTCBench-Wild samples rendering scenes and questions from dynamic pools to simulate visually diverse input scenarios.The rendering pool contains 99 CSS configurations, and each instance is randomly rendered using one of those styles.Source: §5 · p. 13, §5.1 · p. 13

5 VTCBench-Wild

  • VTCBench-Wild was introduced because rendering parameters such as font size, color, and line height can affect VLM performance, making a single fixed rendering configuration insufficient to assess robustness.Source: §5 · p. 13

5.3 Experimental Results and Analysis

  • Only Gemini-3-Pro scores above 50% on VTC-Memory, while strong text perception often fails to yield deep contextual understanding.VTCBench-Wild is presented as a diagnostic tool for architectural resilience and model failure points in realistic scenarios.Source: Table 7 · p. 14, §5.3 · p. 15

5.1 Construction of VTCBench-Wild

  • Retrieval and reasoning questions vary haystack length across 1k–32k tokens and needle depth from 0% to 100% in 10% increments.The memory set uses 1,540 question-answer pairs from the full VTC-Memory dataset; the final Wild benchmark samples 600 memory, 800 reasoning, and 800 retrieval items.Source: §5.1 · p. 14

5.2 Evaluation Metrics

  • Retrieval and reasoning answers are scored with containsAll, while GPT-4o-mini [25] judges memory-answer correctness.Source: §5.2 · p. 14

5.3 Experimental Results and Analysis

  • Gemini-3-Pro VLM scores 87.64% overall, approaching its text-only counterpart at 89.29% and exceeding Qwen3-8B’s 73.55%.Most VLMs fail to match the text-only baselines, while Gemini-3-Pro is a notable exception.Source: §5.3 · p. 15
  • Gemini-2.5-Pro scores 63.68% overall and leads retrieval at 87.38%, while GPT-5 scores 61.23% overall and edges reasoning at 62.88%.The passage reports markedly lower performance for other proprietary models, particularly on reasoning.Source: §5.3 · p. 15
  • Glyph scores 57.27% overall and Qwen2.5-VL-72B scores 57.36%, while GLM-4.1V-9B-Thinking leads open-source VLMs on reasoning at 43.13%.The passage characterizes Glyph and Qwen2.5-VL-72B as balanced performers and GLM-4.1V-9B-Thinking’s reasoning score as competitive with leading proprietary models.Source: §5.3 · p. 15

6 Conclusion

The study finds that VLMs can retrieve text from VTC images, but performance degrades with longer contexts and higher compression, while associative reasoning and long-term memory remain major weaknesses. It concludes that VTC is not a drop-in long-context solution and calls for research into objectives and architectures that bridge visual perception and long-range semantic reasoning.

6 Conclusion

  • VLMs show some simple retrieval ability under VTC, but performance is fragile as context length and compression ratios increase, with associative reasoning and long-term memory especially weak.The conclusion describes a near-total collapse in these two capabilities and reports that Gemini-3-Pro with VTC outperforms the Qwen3-8B LLM baseline on VTCBench-Wild.Source: §6 · p. 15
  • VTC is not a drop-in long-context solution because its efficiency gains come at the cost of advanced cognitive capabilities and visual robustness.Source: §6 · p. 15
  • Effective VTC-based models will require targeted pre-training objectives and architectures that bridge spatial perception with abstract, long-range semantic reasoning.The conclusion positions VTCBench as an evaluation framework intended to guide development of such models.Source: §6 · p. 16

A VTCBench Description

VTCBench evaluates VLM long-context understanding when long text is compressed into images, using retrieval, associative reasoning, and dialogue-memory tasks.

A VTCBench Description

  • VTCBench is a benchmark for VLM long-context understanding with compressed text images, covering information retrieval, associative reasoning, and long-term dialogue comprehension.Source: A VTCBench Description · p. 17

A.1 Rendering Pipeline

VTCBench renders text through a browser-based pipeline and converts PDF pages into images. The rendering preset affects compression: larger font size lowers the number of text tokens represented per image.

A.1 Rendering Pipeline

  • The rendering pipeline interprets Markdown as HTML, renders it to PDF in a browser engine with injected CSS, then converts PDF pages into images.The figure identifies markdown-it, Playwright, and PyMuPDF as the pipeline components.Source: Figure S1 · p. 17, §A.1 · p. 17
  • For VTC-Memory rendered with Rconversation, context length and image count have an inferred correlation of approximately 2,000 tokens per image.Source: Figure S2 · p. 17
  • VTC-Reasoning with Rplain averages 2,000 text tokens per image, versus 1,100 with Rplain-16 using 16-pixel font size.The passage says this nearly halves the VTC compression ratio.Source: §A.1 · p. 17

A.2 Examples of VTCBench

VTCBench examples are generated by varying task, context length, template, and needle type, then rendering sampled contexts into images with prompts. The appendix also calculates compression costs across models’ visual-processing pipelines.

A.2 Examples of VTCBench

  • Configurations vary task type, context length, evaluation template, and needle type; sampled keys, values, and distractors fill the haystack before image rendering and prompt concatenation.Retrieval and reasoning use randomly generated queries, keys, values, and distractors, while memory examples use them as-is.Source: §A.2 · p. 17, Table S1 · p. 18

B Calculation of Compression Ratio

  • The VTC ratio compares text-token count with the image-token count required, and depends on both rendering operator R and the VLM’s visual-processing pipeline.The calculation examples assume an image of size (H, W) and ignore model-specific resizing and padding.Source: B Calculation of Compression Ratio · p. 19

B.1 Dynamic Resolution Models

  • For an 896 × 896 image, Qwen2.5-VL uses 1024 visual tokens, Qwen3-VL uses 784, and Kimi-VL-A3B-Instruct uses 1024.Qwen2.5-VL and Kimi-VL use H×W/(14×14×4); Qwen3-VL uses H×W/(16×16×4).Source: §B.1 · p. 19

B.2 Thumbnail-based Models

  • InternVL3.5 uses 1280 tokens for an 896 × 896 image by combining four 256-token tiles with a global thumbnail.GPT-5’s pipeline yields 630 tokens per image after four 140-token tiles and 70 base tokens.Source: §B.2 · p. 19

B.3 Others

The appendix gives token-count examples for additional image-processing pipelines, including Gemini-2.5-Pro, Deepseek-OCR, and Gemma3.

B.3 Others

  • An 896 × 896 image costs 1032 tokens for Gemini-2.5-Pro, which tiles images into 768 × 768 pixel tiles at 258 tokens each.Table S2 lists inference parameters for the VLMs and LLM baseline.Source: §B.3 · p. 20, Table S2 · p. 20
  • Deepseek-OCR uses the Gundam model, with 656 visual tokens per image.The stated count is 4 × 100 + 256.Source: §B.3 · p. 20
  • Gemma3 resizes images to 896 × 896 pixels and applies 4 × 4 average pooling, producing 256 tokens per image.The passage characterizes this pooling as a high-factor compression strategy.Source: §B.3 · p. 20

C Experimental Details

Experiments use default HuggingFace generation settings with minimal thinking, alongside prompts that set answer formats and grading criteria. The prompt variants include retrieval-style answers, memorized-number questions, and answer checking.

C Experimental Details

  • Experiments follow HuggingFace’s default generation configuration and use the minimal thinking configuration possible.Table S2 presents the inference parameters for VLMs and the LLM baseline.Source: C Experimental Details · p. 20

D Prompts

  • Several prompts ask for a short phrase based on the context, using exact context wording whenever possible.Other answer prompts limit responses to explicitly stated facts or strong logical inferences.Source: D Prompts · p. 21
  • A memory-style prompt asks the model to memorize hidden special numbers in the context before being quizzed about them.The prompt requests a short answer based on the book snippet.Source: D Prompts · p. 21
  • A separate grading prompt compares candidate answers with a supplied standard answer and requires only a CORRECT, INCORRECT, or INVALID judgment.It instructs graders to ignore reasoning and evaluate the final answer against the standard answer.Source: D Prompts · p. 21

E Limitation

The limitations concern language and content coverage, evaluation scope, and opaque proprietary-model APIs. The appendix also identifies retrieval result tables and needle-depth accuracy figures, noting that blank figure cells indicate out-of-memory cases.

E Limitation

  • The benchmark is limited to English and general-domain prose and conversations; other languages and specialized or structured content remain unassessed or may behave differently.The stated examples include Chinese and Arabic scripts, source code, legal documents, and scientific papers.Source: E Limitation · p. 22
  • VTCBench is a snapshot, so its performance findings may not generalize to future VLMs with novel visual-processing or attention mechanisms.The authors call for continuous integration and evaluation of new models.Source: E Limitation · p. 22
  • Proprietary-model APIs can obscure preprocessing and alter rendered images; input-image limits can also affect performance.The passage gives Gemini-2.5-Pro as an example of an API with a maximum-image constraint.Source: E Limitation · p. 22
  • Despite the stated limitations, the authors describe VTCBench as a foundation and rigorous evaluation framework for developing more efficient and capable long-context VLMs.Source: E Limitation · p. 22
  • Tables S3–S6 report VTC-Retrieval results for S-NIAH, MK-NIAH, MV-NIAH, and MQ-NIAH.Source: Table S3 · p. 23, Table S4 · p. 23, Table S5 · p. 24
  • Figures S5–S8 show accuracy against needle-placement depth for MK-NIAH, MV-NIAH, MQ-NIAH, and VTC-Reasoning; blank cells indicate OOM.Figure S6 covers S-NIAH.Source: Figure S4 · p. 25, Figure S5 · p. 26, Figure S6 · p. 27

Related papers

Loading 2512.15649v2…