PaperlayerView exact revision on arXiv

Source-linked AI summary

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Kaiser Sun, Xiaochuang Yuan, Hongjun Liu, Chen Zhao, Cheng Zhang, Mark Dredze, Fan Bai

arXiv:2603.09095v3cs.CLcs.CV

TL;DR

MLLMs often perform worse when identical text is presented as pixels, but existing evaluations do not clearly separate rendering artifacts from perceptual or reasoning limitations. The paper evaluates diverse models, tasks, input modes, and document types, analyzes over 4,000 errors, and applies self-distillation using text-mode reasoning traces. It finds that shortened reasoning chains are the primary supported explanation and that lightweight fine-tuning can close the image-text performance gap.

  • Problem

    Existing work does not systematically explain why the modality gap arises, when visual input helps or hinders, or how to close it with minimal intervention.

  • Method

    The paper evaluates seven MLLMs across seven benchmarks and five input modes, separates reading from reasoning with OCR pipelines, analyzes over 4,000 errors, and fine-tunes models on their own text-mode reasoning traces paired with images.

  • Results

    The modality gap is task- and rendering-dependent, while image input primarily triggers reasoning collapse; self-distillation raises image-mode accuracy to match or exceed text-mode performance.

  • Takeaways & Limitations

    Visual-text performance differences partly reflect distributional mismatch and reduced reasoning effort, making lightweight self-distillation a practical way to improve image-mode understanding.

  • Takeaways & Limitations

    The evidence for reasoning collapse is correlational and intervention-based rather than fully causal, and experiments do not cleanly separate distributional, perceptual, and behavioral contributions.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50\% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.

1 Introduction

The paper systematically diagnoses why MLLMs underperform when text is presented as pixels and proposes a lightweight method that narrows this modality gap.

  • Existing work documents or circumvents the modality gap without systematically explaining its causes, conditions, or minimal-intervention remedies.
  • The evaluation spans seven MLLMs, seven benchmarks, five input modes, and both synthetic renderings and realistic document images.OCR-based modes separate visual reading from downstream reasoning.
  • The gap depends strongly on task, data, font, and resolution, with synthetic math drops exceeding 60 points and font changes shifting accuracy by up to 47 percentage points.Natural document images sometimes produce smaller gaps or even outperform text input.
  • Grounded-theory analysis of over 4,000 errors identifies reasoning collapse—not impaired perception or knowledge retrieval—as the major cause of image-mode degradation.Image inputs produce shorter outputs and increase calculation and format errors while conceptual and reasoning errors remain unchanged.
  • Self-distillation pairs image inputs with the model’s own text-mode reasoning traces, improving image-mode accuracy to match or exceed text-mode performance without architectural changes or external data.The method uses lightweight training and is designed to close the gap with minimal intervention.

2 Related Work

Prior work develops visual-text systems and measures text-versus-image differences, while this paper focuses on diagnosing the failure mechanism and closing the gap in existing MLLMs.

  • Visual-text research includes screenshot language models, vision-centric tokenization, and unified image-text architectures that build systems for processing language visually.
  • Recent evaluations establish task-dependent modality gaps and study cross-modal consistency, but generally do not explain why inconsistencies arise.
  • This paper extends evaluation to realistic Wikipedia and arXiv document images rather than relying only on synthetic renderings.
  • Existing gap-closing approaches either propose no remedy, require training from scratch, or use different forms of cross-modal distillation.The paper instead trains each model on its own text-mode reasoning traces paired with image inputs.

3 Evaluation Setup

The evaluation compares textual, visual, instruction-plus-image, and two OCR pipelines across synthetic and natural document datasets using standardized task metrics.

  • Datasets: The benchmark suite includes synthetic MMLU, ARC, GPQA, GSM8K, and HumanEval tasks, plus natural-image QASPER and SQuAD v2 datasets.Synthetic counterparts render otherwise textual tasks as images; natural contexts come from arXiv PDFs and Wikipedia screenshots.
  • Datasets: QASPER uses arXiv PDF pages, while SQuAD v2 uses screenshots of corresponding Wikipedia pages to create natural-image contexts.
  • Metrics: The study reports each dataset’s standard metric, using pass@k for HumanEval and accuracy for the remaining tasks, with LLM judging for QASPER and SQuAD.
  • Input modes: Five input modes compare Pure Text, Pure Image, Instr.+Image, OCR-1P, and OCR-2P, separating visual reading from reasoning.Pure Text is the reference modality; OCR-2P extracts text first and then solves a Pure Text query.
  • Input modes: Pure Image renders identical textual content into a 1280×720 image, while Instr.+Image keeps task instructions in text.
  • Models: The model pool contains seven MLLMs spanning architectures, scales, and access types, including six open-source models and proprietary GPT-5.2.
  • Evaluation tables: Table 1 covers four synthetic-image datasets and encodes performance relative to Pure Text through red-for-better and blue-for-worse cell shading.HumanEval results are separated because its small dataset produces high pass@1 variance.
  • Evaluation tables: Table 2 compares modalities on QASPER and SQuAD v2, with image-based inputs outperforming Pure Text on QASPER.

4 Data Characteristics and Rendering Confounds

Performance differences depend strongly on task, document distribution, and rendering choices: synthetic images can create large gaps, while natural documents often reduce or reverse them. Font and resolution are major confounds, and OCR-based modes reveal that recognition quality alone does not explain image-mode performance.

  • 4.1 Modality Gap on Synthetic and Natural Images: GSM8K shows drops exceeding 60 points for several models, whereas natural-document QASPER images improve performance for almost all models.Synthetic MMLU and GPQA gaps are typically 1–8 points, while QASPER uses naturally occurring documents and often reverses the text-image trend.
  • 4.2 Image Recognition Cannot Fully Explain the Modality Gap: OCR-1P often fails catastrophically, while OCR-2P can recover performance on GSM8K but destroy code generation by removing structural cues.InternVL3-8B rises from 42.53 to 87.87 on GSM8K with OCR-2P, yet multiple models score zero on HumanEval.
  • 4.2 Image Recognition Cannot Fully Explain the Modality Gap: Overall OCR quality poorly predicts image-mode accuracy, with a correlation of only 0.238 across models and tasks.Task-critical tokens such as operators and variable names may matter more than aggregate recognition accuracy.
  • 4.3 Rendering Choices Confound Visual Evaluation: Font choice swings accuracy by up to 47 percentage points, with handwriting producing the largest degradation.Fonts common in pretraining, including inverted and monospaced styles, cause minimal harm, while handwriting is more out-of-distribution.
  • 4.3 Rendering Choices Confound Visual Evaluation: Most models maintain accuracy at 0.50×–1.0× resolution but degrade sharply below that threshold.InternVL3.5 is uniquely resolution-invariant, and a compact 10pt rendering can outperform higher-resolution counterparts.
  • 4.4 Error Taxonomy: The error taxonomy summarizes grounded-theory coding of 4,195 errors across all models and datasets.The figure reports overall category frequencies rather than a single benchmark score.

5 Error Analysis

Grounded-theory analysis of 4,195 errors shows that image input suppresses extended reasoning rather than removing knowledge or logical capability. Shorter image-mode responses increase calculation and formatting failures, especially on tasks requiring multi-step computation.

  • 5.1 Error Analysis Method: The analysis classifies up to 150 errors per model–dataset pair, yielding 4,195 errors across 33 pairs.The taxonomy was derived through grounded-theory coding and validated with moderate to substantial inter-annotator agreement, κ = 0.57–0.69.
  • 5.1 Reasoning Collapse: Models produce substantially shorter reasoning in image mode, often skipping step-by-step computation or exiting reasoning early.Qwen3-VL-8B averages 618 characters in text mode versus 32 in image mode; Pixtral averages 115 versus 23.
  • 5.2 Error Taxonomy and Findings: Calculation errors increase by 1.5× under image input, from 11.1% to 16.7%, while format errors rise from 5.9% to 8.0%.These are error types that intermediate computation and longer outputs can help prevent.
  • 5.2 Error Taxonomy and Findings: Conceptual/factual recall errors and reasoning errors remain virtually unchanged between image and text modes.The asymmetry indicates suppressed reasoning effort rather than impaired knowledge or logical capabilities.
  • 5.2 Error Taxonomy and Findings: GSM8K errors are dominated by calculation failures, while HumanEval has 36.3% incomplete responses and 24.5% format errors.The dataset profiles reflect task requirements and code generation’s sensitivity to output truncation.
  • 5.3 Implication: Restoring long reasoning traces through self-distillation substantially closes the modality gap.The intervention follows the diagnosis that image mode suppresses step-by-step reasoning and causes cascading calculation and formatting errors.

6 Bridging the Modality Gap via Self-Distillation

The paper restores visual-input reasoning by pairing image inputs with models’ own correct text-mode reasoning traces. Lightweight self-distillation substantially closes the modality gap, transfers to unseen benchmarks, and preserves text-mode performance.

  • Method: Self-distillation trains models on their own text-mode reasoning traces paired with corresponding image inputs.The method transfers multi-step reasoning rather than only final answers and requires lightweight fine-tuning without architectural changes.
  • Results: Across four configurations, self-distillation reduces the text–image gap by an order of magnitude or more.The configurations span Qwen3-VL-8B and InternVL3-8B trained on GSM8K or MMLU.
  • Results: 30.71% to 92.72%: Qwen3-VL-8B trained on GSM8K reaches image-mode accuracy matching text-mode performance.This is the largest reported improvement among the four configurations.
  • Results: Self-distillation transfers to unseen benchmarks while models preserve or improve held-out performance and avoid catastrophic forgetting.For example, InternVL3-8B trained on GSM8K improves MMLU image-mode accuracy from 33.93% to 68.00%.
  • Ablations: LM-only adaptation achieves 92.72%, matching ViT+LM at 92.57%, while ViT-only adaptation reaches 85.29%.The ablation identifies language-model adaptation as the key factor, although visual-feature alignment alone also yields substantial gains.

7 Conclusion

The conclusion frames the modality gap as dependent on rendering and reasoning behavior rather than uniformly limited visual understanding. It also situates the evaluation across diverse reasoning, knowledge, coding, and document-reading tasks.

  • Conclusion: Across seven benchmarks, seven models, and five input modes, the paper diagnoses why text and image inputs produce different performance.The evaluation includes synthetic renderings, realistic document images, and OCR-based configurations.
  • Conclusion: Synthetic renderings are the main difficulty, while natural document images often produce strong performance; font, resolution, and compression can shift accuracy by tens of percentage points.The conclusion attributes these patterns to visual properties that diverge from pretraining distributions.
  • Evaluation Tasks: The evaluation suite spans knowledge, scientific reasoning, mathematics, coding, long-context document comprehension, and extractive question answering.MMLU covers 57 subjects; ARC, GPQA, GSM8K, HumanEval, QASPER, and SQuAD v2 provide complementary task demands.
  • Evaluation Composition: The five synthetic-image benchmarks contain 16,895 instances, while QASPER and SQuAD v2 use natural images separately.MMLU accounts for 83.0% of the synthetic-image benchmark instances.

D Experiment Setting Details

The experiments use a unified decoding setup with task-specific generation limits and evaluate benchmark examples through the paper’s documented image formats. Dataset composition is summarized separately for the evaluation suite.

  • Decoding Setup: Unless otherwise noted, decoding uses temperature 0.1, batch size 8, and a maximum generation length of 1024 tokens.Code generation uses temperature 0.2 to provide diversity.
  • Decoding Setup: OCR-1P uses a 4096-token maximum because one inference must include both OCR output and task reasoning.The longer limit allows full inclusion of extracted text and reasoning.
  • Dataset Composition: The evaluation suite contains five synthetic-image benchmarks and separately lists QASPER and SQuAD v2 as natural-image benchmarks.The synthetic benchmarks are MMLU, ARC, GPQA, GSM8K, and HumanEval.

E HumanEval Results

HumanEval results require cautious interpretation because pass@1 is unstable with only 164 instances and code generation is sensitive to formatting. Resolution sensitivity is generally stable above a lower bound, with InternVL3.5-8B remaining stable across all tested resolutions.

  • HumanEval Evaluation: 164 instances make HumanEval pass@1 estimates statistically unstable.The paper therefore reports HumanEval separately and notes recommendations to generate many samples with an unbiased estimator.
  • HumanEval Evaluation: Code generation is especially vulnerable to formatting failures, while OCR methods can remove indentation and whitespace needed for code comprehension.These structural cues are critical for interpreting and generating code.
  • Resolution Sensitivity: Most models remain stable until resolution reaches a lower bound, while InternVL3.5-8B remains stable across all tested resolutions.The paper attributes the latter robustness to its Visual Resolution Router.
  • Metrics: HumanEval is evaluated with pass@1, while the other listed benchmarks use Accuracy or F1.The complete comparison covers PureText, PureImage, instruction-plus-image, and OCR-based variants across seven benchmarks.

H Evaluation for QASPER and SQuAD v2.

The evaluation uses model-based accuracy for free-form answers and compares performance across input modalities. HumanEval results are highly variable because the benchmark contains only 164 instances.

  • Evaluation metrics: GPT-5 assigns QASPER and SQuAD predictions scores of 1.0, 0.5, or 0.0 against reference answers, with accuracy reported as the mean.This model-based evaluation accommodates free-form, paraphrastic, and multi-sentence answers better than exact-match or span metrics.
  • Benchmark caveat: HumanEval pass@1 results show high variance because the dataset contains only 164 instances.The table caption attributes this instability to the small sample size and pass@1 estimation.
  • Benchmark caveat: OCR-2P substantially reduces performance for several models, suggesting that OCR removes indentation and whitespace cues needed for code comprehension.

I Compute Resources

The paper reports the compute used for evaluation, self-distillation training, and error analysis, including GPU-hour and API-cost estimates.

  • Evaluation experiments: Approximately 1000 GPU-hours supported inference across seven models, seven benchmarks, and five input modes.Most compute went to QASPER, SQuAD, larger models, and multi-stage inference.
  • Self-distillation training: Each self-distillation run used LoRA fine-tuning on one H100 GPU, requiring approximately 2 hours on GSM8K and 3 hours on MMLU.Both datasets contained approximately 7K examples and training lasted two epochs.
  • Error analysis: Coding 4,195 errors with GPT-5.2 API calls cost approximately $500, including preliminary experiments.

J FLOPs Analysis

The paper analyzes visual-token compute, OCR quality, and reasoning behavior while using grounded-theory annotation to characterize errors across modalities.

  • FLOPs analysis: The visual-to-text prefill-FLOPs ratio varies considerably across models and datasets, measuring the computational overhead of image inputs.A higher ratio indicates proportionally greater compute demand from the image pathway.
  • OCR analysis: OCR quality correlates weakly with task accuracy, with r = −0.279 for CER and r = −0.238 for WER.The relationship is not deterministic, so OCR quality alone does not explain accuracy differences.
  • Error annotation: Human annotators reviewed proposed codes, refined the taxonomy, and grouped codes through axial coding after saturation.
  • Error annotation: Three-way annotator agreement was 68.2%, while human–model agreement reached κ ≥0.57, indicating consistent but not ambiguity-free classifications.Remaining disagreements included boundaries between conceptual and reasoning errors.

N Limitations

The paper limits its conclusions through methodological ambiguity, restricted model and benchmark coverage, zero-shot evaluation, and statistical uncertainty in some experiments.

  • Causality: The reasoning-collapse diagnosis is supported by correlational and intervention-based evidence, while alternative mechanisms are not cleanly separated.The authors leave disentangling distributional, perceptual, and behavioral contributions to future work.
  • Scope: The model landscape is rapidly evolving, and newer models already show smaller modality gaps than some evaluated systems.The results therefore represent a snapshot of seven models at a particular point in time.
  • Scope: The evaluation covers seven models and reasoning-intensive benchmarks, so findings may not generalize to other fusion architectures or tasks such as visual grounding.
  • Statistical reporting: HumanEval uses single-sample pass@1 over 164 problems, making its results illustrative rather than precise capability measurements.Self-distillation experiments also use single training runs because multiple seeds were not run.
  • Evaluation setting: All experiments use zero-shot evaluation, leaving the effects of few-shot prompting and chain-of-thought demonstrations unresolved.Such prompting may mitigate reasoning collapse while adding confounding factors.
  • Methodological limitations: The grounded-theory analysis has moderate human–GPT-5.2 agreement of κ = 0.57 and limited fine-grained per-cell statistical support.The authors consider aggregate findings more reliable than per-cell error-distribution claims.
Loading 2603.09095v3…