PaperlayerView exact revision on arXiv

Source-linked AI summary

Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs

Yanhong Li, Zixuan Lan, Jiawei Zhou

arXiv:2510.18279v2cs.CLcs.AI

TL;DR

Long-input inference is costly, and the efficiency–performance trade-off of reading text as images remains under-examined. The paper renders long contexts as images for multimodal LLMs, then evaluates retrieval and summarization. It reports preserved task performance with nearly two-fold decoder-token reductions.

  • Problem

    The efficiency and downstream performance of using text images to reduce input tokens had not been systematically examined, despite the cost of long-input inference.

    Source: §1 · pp. 1–2

  • Method

    The paper renders long contexts as images so an off-the-shelf multimodal LLM processes them as visual tokens, without fine-tuning or supervision.

    Source: §1 · p. 2

  • Results

    Across RULER retrieval and CNN/DailyMail summarization, task performance is preserved with nearly two-fold decoder-token reductions; RULER reaches 97–99% accuracy with up to 58% fewer decoder tokens.

    Source: §1 · p. 2, §5 · p. 5

  • Takeaways & Limitations

    Off-the-shelf multimodal LLMs can treat images as an implicit compression layer without parameter updates, and may lower latency on large decoders.

    Source: §1 · p. 2, §5 · p. 5

    The approach is not fully evaluated on contexts spanning tens of thousands of tokens or more, and experiments cover a limited number of benchmarks, domains, and tasks.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Large language models (LLMs) and their multimodal variants can now process visual inputs, including images of text. This raises an intriguing question: can we compress textual inputs by feeding them as images to reduce token usage while preserving performance? In this paper, we show that visual text representations are a practical and surprisingly effective form of input compression for decoder LLMs. We exploit the idea of rendering long text inputs as a single image and provide it directly to the model. This leads to dramatically reduced number of decoder tokens required, offering a new form of input compression. Through experiments on two distinct benchmarks RULER (long-context retrieval) and CNN/DailyMail (document summarization) we demonstrate that this text-as-image method yields substantial token savings (often nearly half) without degrading task performance.

1 Introduction

Long-input inference is costly, and the efficiency of representing text as images while preserving downstream performance remains under-examined. The paper studies this approach and reports substantial token savings across retrieval and summarization benchmarks.

  • Transformer self-attention scales quadratically with input length, making long-input inference costly and constraining deployment throughput and per-token cost.Source: §1 · p. 1
  • Although multimodal LLMs can read text in images, their efficiency and downstream performance for this use remain under-examined.Source: §1 · pp. 1–2
  • On RULER, GPT-4.1-mini and Qwen2.5-VL sustain 97–99% accuracy with up to 58% fewer decoder tokens; on CNN/DailyMail, the approach beats two pruning baselines at matched or higher compression.Vision processing adds overhead on smaller models, while shorter decoder sequences yield up to 45% end-to-end speedup on larger ones.Source: §1 · p. 2

2 Related Work

Prior work establishes that multimodal models can interpret text in images, while context-compression methods span token selection, summarization, and visual representations. This paper focuses on the efficiency–performance trade-off of using visual text as compression.

  • Multimodal models combine a vision encoder, projection layer, and text decoder; their visual-token sequences can be far shorter than text tokens representing the same information.Source: §2 · p. 2
  • Earlier text-as-image studies show feasibility, but had not systematically tested token-use versus reasoning-performance trade-offs on multi-step tasks.Source: §2 · p. 2
  • Unlike hybrid prompting work focused on multimodal novelty or screenshot coding, this work emphasizes efficiency gains from replacing text with images.Source: §2 · p. 2
  • Context-compression alternatives include dense embedding replacement, query-aware filtering, and token pruning; this approach instead offloads text spans to an off-the-shelf model’s vision encoder.Source: §2 · p. 2
  • At fixed visual-token budgets, both models lose accuracy as rendered text density rises, while Qwen sustains higher text-to-visual-token ratios before a sharper drop.Source: Figure 2 · p. 3

3 Methodology

The method compares text-only context-and-query inputs with hybrid inputs that render the context as an image and retain the query as text. It evaluates accuracy, token usage, throughput, and latency across the two modes.

3.1 Problem Formulation

  • The text-only baseline concatenates context c and query q, using an input budget of m + |q| tokens.Source: §3.1 · p. 3
  • For hybrid input, the context is rendered as an image with layout and line breaks preserved, then encoded into visual tokens projected into the decoder alongside the query.Source: §3.1 · p. 3
  • Visual-token count k depends on model architecture; experiments use each model’s native vision module without fine-tuning or additional supervision.Source: §3.1 · p. 3
  • The compression ratio ρ compares text-only and hybrid token budgets, with k ≪ m possible in practice and higher ρ indicating greater token savings.Source: §3.1 · p. 3

3.2 Evaluation Protocol

  • Each example is evaluated in text-only and hybrid modes for task accuracy, token usage and compression ratio, throughput, and wall-clock latency; with fixed k, reducing m lowers decoder token cost.Source: §3.2 · p. 3

4 Experiments and Results

The experiments test whether rendering text as images reduces decoder tokens without degrading retrieval or summarization, using GPT-4.1-mini and Qwen2.5-VL-72B. Across both tasks, the method preserves retrieval accuracy and outperforms text-pruning baselines on summarization.

4.1 Long-Context Retrieval

  • RULER S-NIAH hides one target number in a long distractor passage and measures exact extraction accuracy over 100 random passages per model.With a short query, the compression ratio is defined as ρ = m/k.Source: §4.1 · p. 4
  • At k = 783, GPT-4.1-mini tolerates m⋆≈1,300 tokens (ρ≈1.9) without measurable degradation; beyond m⋆, Qwen2.5-VL drops more steeply.The tolerance threshold is the largest text length within three percentage points of the text-only baseline.Source: §4.1 · p. 4
  • As visual budget k rises, m⋆ increases at roughly steady ρ≈2; Qwen2.5-VL-72B has a better compression ratio than the 7B model and GPT-4.1-mini.Figure 3 plots text-token tolerance against visual-token count and reports a roughly half reduction in decoder tokens.Source: Figure 3 · p. 4, §4.1 · p. 4

4.1 Long-Context Retrieval; 4.2 Document–Level Summarization

  • Across RULER and CNN/DailyMail, image-based text preserves retrieval accuracy with nearly half the decoder context and beats both pruning baselines on summarization at similar compression rates.The summarization setup retains about 40% of the original tokens.Source: §4.1 · p. 4, Table 1 · p. 5, Table 2 · p. 5

4.1 Long-Context Retrieval

  • Qwen2.5-VL inference is 25–45% faster, while GPT-4.1-mini incurs under 1.5s of vision-processing overhead.The paper attributes Qwen’s faster inference to shorter decoder sequences outweighing vision-processing cost.Source: §4.1 · p. 4, Table 1 · p. 5

4.2 Document–Level Summarization

  • For CNN/DailyMail, image resolution is set from average input length, and Select-Context and LLMLingua-2 are compared at matched decoder compression using ROUGE and BERTSCORE.The two text-only baselines retain tokens using self-information thresholds or a trained token-retention predictor.Source: §4.2 · pp. 4–5
  • The authors propose pruning text tokens before rendering as a future way to compound compression gains.They also suggest testing domains such as math, where many prompt tokens are critical and difficult to prune.Source: §4.2 · p. 5

5 Conclusion

The paper concludes that visual text can nearly halve decoder-token use while preserving performance on the tested retrieval and generation benchmarks. It presents modality shifting as a complementary efficiency direction and proposes combining it with token pruning or extending it to other domains.

5 Conclusion

  • Visual text inputs nearly halve decoder tokens while preserving task accuracy on RULER S-NIAH retrieval and CNN/DailyMail generation.The conclusion describes the approach as model- and task-agnostic and requiring no parameter updates.Source: §5 · p. 5
  • The authors propose applying token-level pruning before visual rendering to further compound compression gains.Source: §5 · p. 5
  • The authors propose exploring domains such as math, where most prompt tokens are critical and difficult to prune at the token level.Source: §5 · p. 5

Limitations

The study has not fully evaluated text-as-image prompting at extremely large context lengths, and its evidence comes from a limited set of benchmarks.

Limitations

  • Text-as-image prompting remains unevaluated for extremely large contexts spanning tens of thousands of tokens or more.The reported token savings concern short- to medium-context scenarios.Source: Limitations · p. 6
  • Document analysis and long conversational histories may require chunking or retrieval for reliability at larger scales.These are proposed as specialized approaches for real-world applications.Source: Limitations · p. 6
  • Evidence is limited to a small benchmark set, leaving medical and legal domains and coding and translation tasks open for evaluation.Source: Limitations · p. 6

A Results Across Context Lengths and Image Sizes on RULER

RULER results show that performance depends on balancing visual resolution and text within the context budget: a near-equal allocation preserves accuracy in tested settings, but more context alone does not ensure stability.

A Results Across Context Lengths and Image Sizes on RULER

  • Higher image resolution can add visual tokens and reduce text capacity, while lowering resolution risks losing semantically important details.Small shifts in token allocation can cause large accuracy changes in long-context retrieval settings.Source: A Results Across Context Lengths and… · p. 9
  • With visual tokens at roughly half the total context window, performance remains nearly indistinguishable from the original uncompressed text; ρ = 2 is a practical efficiency–fidelity heuristic.The authors say its robustness across tasks, datasets, and model architectures needs further validation.Source: A Results Across Context Lengths and… · p. 9
  • At context lengths of 1,500, 2,000, and 2,500, representative RULER cases achieved accuracy close to 100%.The reported image resolutions were 600 × 800, 600 × 1,000, and 750 × 1,000 pixels, respectively.Source: A Results Across Context Lengths and… · p. 9
  • At 3,000 tokens with a 600 × 1,000 image, accuracy degraded substantially despite the larger context.The authors attribute stable performance to balanced textual and visual token allocation, not context length alone.Source: A Results Across Context Lengths and… · p. 9
  • GPT-4.1-mini API response times include system overhead, and larger image payloads may partly explain the higher latency for text-as-image inputs.The latency measurements therefore may not reflect pure model latency.Source: A Results Across Context Lengths and… · p. 9

B Results on Qwen2.5-VL-7B-Instruct

On long-context retrieval, Qwen2.5-VL-7B-Instruct tolerates less visually rendered text than the 72B model, with a steeper decline as text density increases.

B Results on Qwen2.5-VL-7B-Instruct

  • With 998 visual tokens, the 72B model maintains over 97% accuracy up to nearly 2,400 text tokens, while the 7B model falls below 95% after 2,000.The 7B model is substantially more sensitive to text density across corresponding visual-token budgets.Source: B Results on Qwen2.5-VL-7B-Instruct · p. 9, Figure 4 · p. 10
  • The comparison identifies model scale as a critical factor for robustly processing dense text presented visually and achieving high compression without performance loss.Source: B Results on Qwen2.5-VL-7B-Instruct · p. 9
  • Figure 4 plots 7B accuracy against rendered text-token size for fixed visual-token budgets, with each curve varying text amount.Compared with the 72B model in Figure 2b, the 7B model degrades more steeply as text density rises.Source: Figure 4 · p. 10

C BABILong 1k Benchmark

On BABILong 1k, performance is stable when image tokens are roughly half the original context text-token count, supporting a near-equal image–text allocation as an operating point.

C BABILong 1k Benchmark

  • BABILong extends bAbI tasks to long contexts with up to 1k distractor tokens, testing whether multimodal token allocation remains effective in long-context reasoning.Source: C BABILong 1k Benchmark · p. 10
  • The experiment uses the standard BABILong 1k variant, ten QA subsets spanning retrieval, coreference, and induction, and Gemini 2.5 Flash Preview.Source: C BABILong 1k Benchmark · p. 10
  • With image tokens at roughly half the original total context text tokens, BABILong 1k achieves accuracies of 0.91 for text and 0.83 for image.The passage reports this pattern across QA1–QA10.Source: C BABILong 1k Benchmark · p. 10
  • The near-equal image–text allocation provides an operating point for multimodal long-context reasoning in this benchmark.Source: C BABILong 1k Benchmark · p. 10

D ConTexImage: A Text-to-Image Conversion Pipeline

ConTexImage renders text as raster images through preprocessing, LaTeX rendering, and adaptive font-size selection. The pipeline aims for readable, consistently rendered inputs with controlled image-token counts.

D ConTexImage: A Text-to-Image Conversion Pipeline

  • ConTexImage renders arbitrary text as images while adjusting font size to meet a target content density.The pipeline is designed to preserve readability and consistent tokenization patterns across resolutions.Source: D ConTexImage: A Text-to-Image… · p. 10
  • The three stages are text preprocessing, LaTeX rendering and rasterization, then adaptive font optimization based on image fill ratio.Preprocessing normalizes typographic symbols and escapes LaTeX characters; the rendered page is resized to the target resolution.Source: D ConTexImage: A Text-to-Image… · p. 10
  • The algorithm selects the largest candidate font size that meets a predefined fill-ratio target, balancing legibility and whitespace.The fill ratio measures the proportion of the image occupied by text.Source: D ConTexImage: A Text-to-Image… · p. 10
  • Nearly no accuracy degradation occurs at contexts 1,500 (600 × 800), 2,000 (600 × 1,000), and 2,500 (750 × 1,000), but degradation is substantial at 3,000 (600 × 1,000).The resulting images provide control over image tokens relative to text tokens.Source: D ConTexImage: A Text-to-Image… · p. 10, Figure 5 · p. 12, Figure 6 · p. 13

Related papers

Loading 2510.18279v2…