PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?

Qing'an Liu, Juntong Feng, Yuhao Wang, Xinzhe Han, Yujie Cheng, Yue Zhu, Haiwen Diao, Yunzhi Zhuge, Huchuan Lu

arXiv:2602.04802v3cs.CV

TL;DR

Existing VLM evaluations largely assume pure-text queries, leaving unclear whether models handle equivalent language rendered as pixels. VISTA-Bench conducts controlled pure-text versus visualized-text evaluations across perception, reasoning, and knowledge, finding a pervasive modality gap that worsens with perceptual difficulty.

  • Problem

    Existing VLM benchmarks predominantly use pure-text queries, leaving the stability of model behavior for language conveyed as visualized text unclear.

  • Method

    VISTA-Bench compares matched pure-text and visualized-text questions across multimodal perception, reasoning, and unimodal knowledge using controlled rendering and fidelity validation.

  • Results

    Over 30 VLMs show a pervasive modality gap, with performance degradation under visualized text that is amplified by perceptually challenging renderings.

  • Takeaways & Limitations

    VISTA-Bench provides a testbed for diagnosing visualized-text limitations and motivates more robust unified representations across tokenized text and pixels.

  • Takeaways & Limitations

    Performance may change as frontier models, APIs, decoding settings, and preprocessing pipelines evolve, requiring continual benchmark updates.

Abstract

from arXiv · show

Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs, yet existing benchmarks predominantly focus on pure-text queries. In real-world scenarios, language also frequently appears as visualized text embedded in images, raising the question of whether current VLMs handle such input requests comparably. We introduce VISTA-Bench, a systematic benchmark from multimodal perception, reasoning, to unimodal understanding domains. It evaluates visualized text understanding by contrasting pure-text and visualized-text questions under controlled rendering conditions. Extensive evaluation of over 30 representative VLMs reveals a pronounced modality gap: models that perform well on pure-text queries often degrade substantially when equivalent semantic content is presented as visualized text. This gap is further amplified by increased perceptual difficulty, highlighting sensitivity to rendering variations despite unchanged semantics. Overall, VISTA-Bench provides a principled evaluation framework to diagnose this limitation and to guide progress toward more unified language representations across tokenized text and pixels. The source dataset and code are publicly available at https://github.com/QingAnLiu/VISTA-Bench.

1 Introduction

VISTA-Bench addresses the underexamined question of whether VLMs understand visualized text as well as equivalent pure text. It introduces controlled comparisons across perception, reasoning, and knowledge, revealing a pervasive modality gap linked to perceptual robustness.

  • Existing VLM benchmarks predominantly use discrete pure-text queries, leaving visualized language and its pixel-level perceptual challenge underexamined.
  • VISTA-Bench compares matched pure-text and visualized-text questions across multimodal perception, reasoning, and unimodal knowledge.It contains 1,500 carefully filtered samples organized under a hierarchical capability taxonomy.
  • Evaluations of over 30 VLMs find a pervasive modality gap that grows with perceptually challenging renderings.The analysis attributes the gap primarily to limited perceptual robustness, while additional visual grounding provides only partial mitigation.
  • The benchmark analyzes visualized-text understanding across distinct multimodal tasks rather than restricting evaluation to unimodal settings.
  • VISTA-Bench frames text-as-pixels as a testbed for diagnosing modality gaps and advancing unified vision-language representations.

2 Related Work

Existing visual-language benchmarks have expanded beyond OCR toward cognition, reasoning, and external knowledge, but they remain predominantly pure-text. VISTA-Bench supplements them with systematic visualized-text evaluation to expose modality differences.

  • Early benchmarks such as DocVQA and OCRBench-v2 focused primarily on localized text recognition and document-level OCR.
  • Later benchmarks broadened evaluation to higher-level visual cognition, reasoning, and integration of external knowledge.
  • Most existing evaluation paradigms still predominantly focus on pure-text inputs despite this broader capability coverage.
  • VISTA-Bench supplements these paradigms with visualized-text evaluation spanning multimodal perception, reasoning, and unimodal understanding.

3 VISTA-Bench

VISTA-Bench systematically compares pure-text and visualized-text processing while controlling rendering and evaluating perceptual factors. Preliminary experiments motivate a benchmark spanning multiple capabilities, with fidelity checked through rendering and review procedures.

  • Task Settings: VISTA-Bench evaluates matched pure-text and visualized-text inputs under unimodal and multimodal tasks, including MMLU, MMBench, Seed-Bench, and MMMU.
  • Preliminary Experiments: 75.99% to 68.46%: Qwen3-VL-8B-Instruct’s MMLU accuracy falls when pure text is replaced by visualized text.
  • Preliminary Experiments: Font size substantially affects visualized-text accuracy, with 32–48pt generally improving performance and 64pt sometimes saturating or declining.
  • Preliminary Experiments: Qwen3-VL-8B-Instruct drops from 68.5% with Arial to 64.5% with Brush on MMLU, while standard fonts show only minor variation.
  • Preliminary Experiments: Prompt effects are model- and task-dependent: moderate semantic guidance can help, while Chain-of-Thought substantially benefits InternVL3.5-8B on some tasks.
  • Preliminary Experiments: Stronger text recognition is associated with a smaller modality gap, with Qwen3-VL-8B-Instruct outperforming InternVL3.5-8B on visualized-text robustness.
  • VISTA-Bench Construction: The construction pipeline filters diverse accurate data, renders text, code, and formulas with specialized handling, and validates fidelity using a VLM judge plus manual review.
  • VISTA-Bench Construction: The benchmark contains 1,500 instances organized into 4 primary tasks, 10 sub-tasks, and 27 fine-grained dimensions.

4 Experiments

Experiments show that visualized-text inputs create a widespread, model-dependent accuracy gap relative to pure text. The gap grows with perceptual difficulty and is especially pronounced for reasoning and knowledge tasks.

  • Main Results: Most VLMs lose accuracy when switching from pure-text to visualized-text inputs, with drops ranging from 0.3 points for MiMo-VL-7B-RL to 31.0 for NEO-9B-SFT.Closed-source models also vary substantially: Gemini-3.1-Pro-Preview drops 1.6 points, whereas GPT-5.2 drops 11.7 points.
  • Fine-grained Analysis: Reasoning and multimodal knowledge degrade more than perception, with Ovis2.5-9B dropping 1.0, 9.4, and 15.7 points across perception, reasoning, and knowledge.Kimi-VL-A3B-Thinking shows a similar pattern, with a 1.3-point perception gap and a 13.3-point reasoning gap.
  • Fine-grained Analysis: Unimodal knowledge exposes the bottleneck: NEO-9B-SFT drops 40.6 points despite strong pure-text accuracy, while several stronger models remain near parity.Qwen2.5-VL-7B-Instruct improves by 0.4 points, and Gemini-3.1-Pro-Preview drops 0.6 points.
  • Fine-grained Analysis: OCR improves visualized-text accuracy but does not eliminate the gap, as PaddleOCR-VL-1.5 reaches 56.0 and 56.9 overall accuracy while remaining below pure-text scores of 59.1 and 59.3.Tesseract performs much worse, reaching only 33.2 and 35.4 overall accuracy for the two evaluated models.
  • Fine-grained Analysis: Rendering difficulty sharply increases the gap: GLM-4.1V-9B-Thinking reaches gaps of 7.7 points for 9 pt text and 9.2 for Brush-style handwriting, versus 3.1 overall.Larger fonts and standard styles can nearly close the gap and occasionally outperform pure text.
  • Fine-grained Analysis: Across 18 models, the average Unimodal Gap is 15.2%, compared with 10.5% for the Multimodal Gap.The reported comparison attributes the lower multimodal gap to contextual visual evidence that can help constrain interpretations.

5 Limitations and Discussion

The benchmark’s conclusions are bounded by rapidly changing frontier models and by its focus on discriminative VLM evaluation. The authors identify agentic auxiliaries and generative models as future directions.

  • Limitations: Performance may change as frontier models receive new versions, API updates, decoding settings, or preprocessing pipelines.The authors recommend continuously updating VISTA-Bench and testing broader real-world deployment settings.
  • Future Directions: The study focuses on VLMs, while agentic pipelines for LLMs and generative or unified multimodal models remain complementary directions for evaluation.Agentic auxiliaries could help disentangle visual recognition, tool-use, and language-side reasoning failures.

6 Conclusion

VISTA-Bench evaluates visualized-text understanding across multimodal and unimodal capabilities using controlled paired inputs. Its 1,500-question dataset spans four task categories and diverse rendered layouts.

  • Dataset Structure: The dataset includes 500 unimodal-knowledge, 400 multimodal-knowledge, 300 multimodal-perception, and 300 multimodal-reasoning instances.These categories target knowledge retrieval, visual-evidence integration, perceptual grounding, and reasoning under visualized text.
  • Dataset Structure: The benchmark contains 1,462 multiple-choice questions and 38 open questions, with the open subset generally more challenging.Every question is rendered as a high-resolution visualized-text image preserving the full textual content in pixels.
  • Rendered Inputs: Question images vary widely in layout: widths are standardized to 800 pixels, heights range from 88 to 7,683 pixels, and aspect ratios range from 0.1 to 9.1.Most images are below 500 pixels in height, while 52.9% have a width-to-height ratio of at least 4.0.

A.3 Benchmark Scale and Stability

The benchmark uses 1,500 filtered samples and a controlled rendering pipeline to balance evaluation stability, cost, and visual diversity. Scale analysis supports the full benchmark as a more stable basis for model comparison.

  • Benchmark Scale: The 1,500-question benchmark is selected as a practical balance among evaluation cost, sample difficulty, and result stability after filtering an approximately 4,000-sample pool.Filtering combines model evaluation, rendering verification, and manual review.
  • Benchmark Stability: Smaller subsets can produce optimistic or composition-dependent estimates: InternVL3.5-8B scores 55.0 VT accuracy on 1,000 samples but 50.2 on all 1,500.The authors therefore treat the full 1,500-question setting as more stable for model-level comparison.
  • Rendering Pipeline: The rendering pipeline converts linguistic tokens into pixel images while preserving typographic fidelity and simulating real-world document characteristics.It includes preprocessing, LaTeX-based generation, rasterization, and post-processing stages.
  • Rendering Pipeline: Rendering configurations standardize image width at 800 pixels and use a DPI of 72.27 while varying font families and sizes from 9pt to 48pt.These settings are intended to maintain reproducibility and typographic diversity across the dataset.

B.3 Sensitivity to Rendering Layout

VISTA-Bench results vary with rendering layout because visualized text is processed as pixels, but these choices do not eliminate the modality gap. The benchmark uses automated rendering checks followed by manual verification for potentially flawed samples.

  • Layout sensitivity: Rendering width and margins are varied one at a time to test whether visualized-text results depend on layout configuration.The default uses width 800 pixels, DPI 72.27, 60-pixel side margins, and 40-pixel top/bottom margins.
  • Layout sensitivity: Smaller widths increase line wrapping and image length, while narrower margins can increase boundary effects or character truncation.Larger widths reduce wrapping but alter text density; wider margins add blank space and reduce the relative textual area.
  • Observed effects: Qwen3-VL-8B-Instruct reaches 50.1%–52.6% visualized-text accuracy versus 59.1% pure-text accuracy across rendering layouts.For InternVL3.5-8B, visualized-text accuracy ranges from 49.1% to 52.1%.
  • Observed effects: Across widths and margins, visualized-text inputs consistently underperform corresponding pure-text inputs, with rendering parameters changing gap magnitude rather than the main conclusion.Pure-text results remain unchanged because they do not depend on rendering.
  • Verification: The VLM rendering-quality judge assigns scores from 0 to 2 for text, code, and formula fidelity, sending samples below 2 to manual verification.The protocol evaluates rendering quality rather than semantic correctness and compares each image with its reference text.
  • Verification: The judge may falsely assign below-perfect scores to valid samples under small-font or dense-layout conditions, so manual review prevents incorrect filtering.The reported cases show inaccurate assessments caused by recognition difficulties under different rendering conditions.

C.2 Additional Preliminary Results

Additional ablations show that rendering sensitivity persists across larger models and standard fonts, while font size and handwritten styling materially affect visualized-text understanding. These results extend the preliminary findings beyond the initial 8B-model setting.

  • Rendering factors: Very small fonts hurt readability, 32–48pt renderings usually improve performance, and overly large text may saturate or slightly degrade.Larger text can induce layout changes such as additional line wrapping.
  • Rendering factors: Standard fonts such as Arial, Times New Roman, and Cambria perform similarly, whereas handwritten Brush Script MT is generally more challenging.At larger sizes, standard-font differences remain relatively small, while Brush Script MT is less stable.
  • Model scale: Qwen3-VL-32B-Instruct and Qwen3.5-397B-A17B remain sensitive to rendering choices, showing that sensitivity also appears in stronger and larger models.The additional study uses MMBench and MMLU to cover multimodal and unimodal task paradigms.
  • Evaluation setup: The evaluation covers open-source VLMs across small, mid-sized, MoE, and larger-scale regimes using official or integrated wrapper configurations.The suite includes families such as InternVL, Qwen-VL, LLaVA, MiMo-VL, GLM, Ovis, SAIL-VL2, and NEO.
  • Evaluation setup: Model outputs are converted into final answers by an API-based extractor that maps multiple-choice responses to option letters without re-solving the question.Refusals, failures, and outputs that cannot be mapped reliably are counted as incorrect.

D.1 Rendering Factor Ablations

Across eight additional representative models, perceptually difficult renderings enlarge the pure-text versus visualized-text performance gap, while cleaner renderings reduce it for some models without removing residual failures.

  • D.1 Rendering Factor Ablations: Very small fonts and Brush-style text produce markedly larger modality gaps across most of the eight additional models.The gap is reported as AccText −AccVT, comparing Pure Text and Visualized Text conditions.
  • D.1 Rendering Factor Ablations: Cleaner renderings substantially reduce the gap for many models, and a small subset becomes nearly gap-free or slightly improves under Visualized Text.Several models nevertheless retain large residual gaps under standard settings, indicating limitations beyond perceptual readability.

D.2 Qwen-Image-Edit Evaluation

Qwen-Image-Edit is evaluated end-to-end by generating visualized questions and answers in designated regions, with humans assessing generation validity and correctness. Performance remains low because both generation stability and pixel-level understanding constrain the task.

  • Evaluation protocol: The 200-instance evaluation samples 50 instances from each of four tasks, rendering questions and options alongside the problem image as one visual input.Human evaluators check whether readable question/answer text is generated in the specified region and whether the final answer is correct.
  • Results: 22.5% overall accuracy is achieved across all 200 samples, compared with 22.15% on the 149 valid generations.The similar accuracies indicate that valid text generation alone does not resolve the downstream task difficulty.
  • Results: 149 generations are valid, 50 are labeled no question, and one sample with minor encoding artifacts is excluded.Valid generations contain readable question/option text in the designated region and a final answer.
  • Interpretation: The results indicate that end-to-end performance is jointly constrained by generation robustness and visualized-text understanding.The evaluation is designed to assess both layout-controlled generation and task performance, which automatic string matching does not robustly capture.
Loading 2602.04802v3…