PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding

Yuling Shi, Chaoxiang Xie, Zhensu Sun, Yeheng Chen, Chenxu Zhang, Longfei Yun, Chengcheng Wan, Hongyu Zhang, David Lo, Xiaodong Gu

arXiv:2602.01785v3cs.CLcs.SE

TL;DR

Text-based code understanding becomes increasingly costly as software systems scale, while evidence on whether code images can serve as an effective alternative remains limited. This paper systematically evaluates multimodal models on visual code understanding and finds competitive or superior performance with image compression using 25% or fewer tokens.

  • Problem

    Evidence remains limited on whether multimodal models can effectively understand and reason over code images instead of textual code, despite growing context-length costs.

  • Method

    The paper systematically evaluates multimodal models across code-understanding tasks using rendered code images, varying compression and visual enhancements such as syntax highlighting.

  • Results

    Across four tasks, visual code achieves comparable or superior performance to text, remains effective at up to 8× compression, and benefits from visual enhancements at moderate compression.

  • Takeaways & Limitations

    Visual code representation is a viable, potentially more token-efficient medium for code understanding, with compression using 25% or fewer tokens sometimes retaining or improving performance.

  • Takeaways & Limitations

    Primary experiments focus on Python, with key findings replicated on Java rather than a broader range of programming languages.

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable success in source code understanding, yet as software systems grow in scale, computational efficiency has become a critical bottleneck. Currently, these models rely on a text-based paradigm that treats source code as a linear sequence of tokens, which leads to a linear increase in context length and associated computational costs. The rapid advancement of Multimodal LLMs (MLLMs) introduces an opportunity to optimize efficiency by representing source code as rendered images. Unlike text, which is difficult to compress without losing semantic meaning, the image modality is inherently suitable for compression. By adjusting resolution, images can be scaled to a fraction of their original token cost while remaining recognizable to vision-capable models. To explore the feasibility of this approach, we conduct the first systematic study on the effectiveness of MLLMs for code understanding. Our experiments reveal that: (1) MLLMs can effectively understand code with substantial token reduction, achieving up to 8x compression; (2) MLLMs can effectively leverage visual cues such as syntax highlighting, improving code completion performance under 4x compression; and (3) Code-understanding tasks like clone detection exhibit exceptional resilience to visual compression, with some compression ratios even slightly outperforming raw text inputs. Our findings highlight both the potential and current limitations of MLLMs in code understanding, which points out a shift toward image-modality code representation as a pathway to more efficient inference.

1 Introduction

The paper investigates image-based code representation as a token-efficient alternative to text, evaluating multimodal LLMs across tasks, compression ratios, rendering strategies, and programming languages. It finds visual code understanding viable, with existing models sometimes matching or surpassing text baselines, while visual enhancements depend on compression conditions.

  • Motivation: Text-based code processing creates context-length and computational-efficiency bottlenecks as software systems grow in scale and complexity.Source code is primarily treated as a linear token sequence, causing associated computational overhead.
  • Experimental scope: The study evaluates seven multimodal LLMs on four tasks across 1×–8× compression ratios and plain, highlighted, and bolded renderings.The tasks are code completion, code summarization, clone detection, and code question answering.
  • Core findings: Visual enhancements can help code-image understanding, but bold rendering may worsen degradation at extreme compression by reducing character clarity.The findings indicate a compression sweet spot for visual enhancements and motivate adaptive rendering strategies.
  • Implementation: The paper implements CodeOCR, a practical renderer supporting configurable visual enhancements and compression ratios for token-efficient code processing.The tool is intended for researchers and developers using LLMs with rendered source code.
  • Core findings: Image-based code representation is viable, with multiple existing LLMs achieving comparable or superior performance to text-based baselines without targeted optimization.The contribution frames this as a technical direction for code understanding.

2 Background

MLLMs process text and images in a unified architecture, enabling source code to be represented as a two-dimensional visual artifact rather than a sequence of text tokens. Their pipeline renders code as an image, encodes and compresses visual patches, fuses them with text prompts, and processes the resulting multimodal sequence.

  • State-of-the-art MLLMs such as GPT-5 and Gemini-3 natively process text and images within a unified architecture.
  • Stage 1: Inputs: Stage 1 renders code as an image with syntax highlighting and indentation, while a text prompt supplies the instruction.Unlike text-based models that tokenize raw code strings, MLLMs treat code as a 2D visual artifact.
  • Stage 2: Encoding & Tokenization: Stage 2 divides the rendered image into fixed-size patches, such as 14×14 pixels, and converts them into visual embeddings.The text prompt is tokenized separately into words or subwords.
  • Stage 3: Alignment & Fusion: Stage 3 applies pooling and projection to compress adjacent visual patches before concatenating aligned visual and text embeddings into one input sequence.A 2 × 2 pooling operation merges four patches while preserving semantic density.
  • Stage 4: Multimodal Modeling & Output: The MLLM backbone processes the unified multimodal sequence through self-attention layers to produce the output.
  • MLLMs interpret continuous visual patterns such as color-coded keywords, indentation depth, and bracket alignment directly from pixels, without explicit parsing.Multimodal variants maintain comparable performance to text-only counterparts on NLP and coding benchmarks, supporting image-based source-code representation.

3 Experimental Setting

The experiments evaluate visual code understanding through five research questions covering effectiveness, compression resilience, visual enhancements, cross-language generalization, and compression-induced errors. The study uses four code-understanding tasks, seven multimodal LLMs, and controlled image-rendering and input-modality settings.

  • Research Questions: Five research questions examine visual-versus-textual effectiveness, compression resilience, visual enhancements, language generalization, and errors caused by visual compression.The experimental design progresses from feasibility comparison to compression, enhancement, generalization, and error analysis.
  • Tasks and Benchmarks: Four representative tasks assess progressively different comprehension capabilities: code completion, summarization, clone detection, and question answering.The study primarily evaluates Python and extends cross-language analysis to Java.
  • Tasks and Metrics: Code completion uses Exact Match (EM) and Edit Similarity (ES), while summarization uses CompScore, clone detection reports Accuracy (ACC) and F1 score, and question answering selects among multiple-choice answers.CompScore ranges from 0–100, with 50 indicating parity; EM measures exact matches and ES captures partial correctness.
  • Models: Seven state-of-the-art multimodal LLMs span proprietary and open-weight categories, including GPT-5-mini, GPT-5.1, Gemini-2.5-Pro, Gemini-3-Flash, Gemini-3-Pro, Qwen-3-VL, and GLM-4.6v.Qwen-3-VL has 235B parameters and GLM-4.6v has 108B parameters; proprietary models integrate multimodality natively.
  • Code Rendering: Source code is rendered into 2240×2240-pixel images, with plain black monospace text on white as the baseline configuration.The base resolution is divisible by common 14- and 16-pixel visual-encoder patch sizes, avoiding partial patches during tokenization.
  • Input Modality: Code content is provided as images while task instructions remain textual, isolating the effect of visual code representation across completion, summarization, clone detection, and question answering.The input design renders retrieved context or code snippets as images while preserving prompts, prefixes, questions, or options as text.

4 Results and Analysis · 4.1 RQ1: How Effective are LLMs in Understanding Visualized Code vs. Textual Code?

Across four Python tasks, code images generally match or outperform raw text, demonstrating the feasibility of visual code representations. Effectiveness varies by model and task: Gemini-3 is most consistent, while some models degrade substantially.

  • 4 Results and Analysis: The study evaluates seven models on four Python tasks, comparing raw text with code images and No Context baselines.The No Context baseline tests whether correct answers reflect genuine code understanding rather than memorization.
  • 4.1.1 Feasibility of Code Image Understanding.: Across all four tasks, code-image inputs achieve comparable or superior performance to raw text.This establishes image-based code representation as a viable alternative to textual input.
  • 4.1.1 Feasibility of Code Image Understanding.: 42% F1 improvement: GPT-5-mini increases clone-detection F1 from 33.2 to 47.0 with code images.The improvement is statistically significant at p< 0.01.
  • 4.1.1 Feasibility of Code Image Understanding.: Gemini-3-Pro performs comparably or better across all four tasks with image-based code representations.Visual inputs may help models perceive indentation, block boundaries, and long-range dependencies holistically.
  • 4.1.2 Model-Specific Variation.: The Gemini-3 family shows the most consistent results, whereas Qwen-3-VL’s code-completion ES drops from 49.7 to 35.5 and GLM-4.6v’s clone-detection accuracy drops from 81.6 to 69.6.The degradation is significant at p< 0.01, and stronger models generally understand code images more effectively.
  • 4.1.2 Model-Specific Variation.: 72.6% accuracy: GLM-4.6v’s image-based code QA performance exceeds its No Context baseline of 37.2%.This supports the conclusion that improved image performance reflects extraction of meaningful visual-code information.
  • 4.1.2 Model-Specific Variation.: Clone detection shows the strongest visual advantage, while code summarization shows no significant difference between modalities.The authors attribute clone-detection gains to visual focus on high-level semantic patterns over token-level syntactic differences.
  • 4.1.2 Model-Specific Variation.: Current LLMs are not fully optimized for code-image understanding, making this gap an important direction for future research.Effectiveness varies by model despite consistent results from the Gemini-3 family.

4.2 RQ2: How Resilient are LLMs to Visual Compression Across Different Coding Tasks?

Compression resilience differs substantially across coding tasks and models: summarization and clone detection tolerate 4×–8× compression better than completion and question answering, while stronger models—especially Gemini-3-Pro—can maintain or improve performance at 8×.

  • Task-level resilience: 58.4 vs. 57.1 raw text: GPT-5-mini significantly improves code summarization at 4× compression.The improvement is statistically significant at p< 0.05.
  • Task-level resilience: 75%: GPT-5-mini’s clone-detection F1 increases from 33.2 raw text to 58.2 at 2× compression.The improvement is significant at p< 0.01.
  • Model-level resilience: Gemini-3-Pro maintains or improves performance at 8× compression, whereas weaker models such as Qwen-3-VL and GLM-4.6v degrade as compression increases.The observations suggest compression resilience correlates with overall model capability in code-image understanding.
  • Task-level resilience: Compression resilience varies by task: summarization and clone detection tolerate higher ratios, whereas completion and question answering degrade beyond 2×–4× compression.Some models maintain performance at 4×–8× compression in summarization and clone detection.
  • Model-level resilience: 58.0 vs. 55.8 raw text: Gemini-3-Pro achieves higher code-completion ES at 8× compression.Across all four tasks, the Gemini-3 family shows no significant degradation and significant improvements in code completion and question answering at 8×.
  • Model-level resilience: 79.5% vs. 74.8% raw text: Gemini-3-Pro improves code question-answering accuracy at 8× compression.At 8× compression, Gemini-3-Pro uses only 12.5% of text tokens, while weaker visual models show pronounced degradation.

4.3 RQ3: Can Visual Enhancements Improve Code Image Understanding?

Visual enhancements, including syntax highlighting and bold rendering, significantly improve code image understanding most at 1×–4× compression, while their benefits diminish at 8× compression. Improvements vary by model and task, and some enhancements can degrade performance when resolution is extremely low.

  • At 1×–4× compression, syntax highlighting and bold rendering significantly improve code image understanding while visual signals remain legible.These enhancements are especially effective for state-of-the-art models, which show consistent 2–5% improvements across tasks.
  • Code completion: At 1× compression, GLM-4.6v improves from ES 50.8 with plain rendering to 53.2 with highlighting, while GPT-5.1 benefits significantly from bold rendering with ES: 50.1 vs. 47.9.Both code-completion comparisons are statistically significant at 𝑝< 0.01.
  • Model responsiveness: Gemini-3-Flash achieves significant improvements with both enhancement strategies at 1×–2× compression.The improvements are statistically significant at 𝑝< 0.05.
  • Task-specific results: At 4× compression, GPT-5.1’s clone-detection F1 improves from 58.2 to 64.6 with bold rendering, a +11% gain, while Gemini-3-Flash reaches 76.8% versus 74.8% for plain rendering in question answering at 1×.The reported improvements are statistically significant at 𝑝< 0.01; Gemini-2.5-Pro also benefits from both strategies at moderate compression.
  • High compression: At 8× compression, visual enhancements generally provide diminishing returns because reduced resolution obscures their visual distinctions.Some combinations remain significant, including Gemini-3-Pro with bold rendering for summarization and Gemini-3-Flash with bold rendering for clone detection; bold can also slightly degrade performance.
  • Adaptive rendering: Enhancement strategies should adapt to compression level, remaining meaningful at moderate compression but potentially unnecessary at high ratios.Adaptive rendering is proposed as a direction for future optimization.

4.4 RQ4: Can Code Image Understanding Generalize to Other Languages?

Experiments on Java show that the core patterns of code image understanding found in Python generalize across languages with different syntax. Visual inputs improve code completion and clone detection, including substantial compressed-input gains for Qwen-3-VL.

  • Code completion: Gemini significantly outperforms raw text for Java code completion across all compression levels (p< 0.01).This result demonstrates strong visual code understanding beyond Python.
  • Clone detection: F1: 24.2 →53.0 at 8×, +119%, for Qwen-3-VL in Java clone detection (p< 0.01).Visual inputs significantly improve clone detection across multiple models, with Qwen-3-VL showing particularly large gains under compression.
  • Cross-language generalization: Java experiments reproduce the fundamental Python patterns, including model-specific strengths and resilience to compression.The study extends code completion and clone detection experiments to Java, whose syntax differs from Python through explicit braces rather than whitespace indentation.

4.5 RQ5: How Does Visual Compression Degrade the Information in Code?

Visual compression degrades code reconstruction through a predictable hierarchy, but model capability determines resilience. Gemini-3 degrades gracefully, preserving downstream performance better than models exhibiting a performance cliff at high compression.

  • Model resilience: At 1× compression, Gemini-3-Pro achieves the highest Exact Match and lowest CER, followed by Gemini-3-Flash and GPT-5.1.This establishes model capability as a determinant of reconstruction resilience.
  • Model resilience: Gemini-3 demonstrates graceful degradation, maintaining high CodeBLEU at 8× compression, whereas other models decline rapidly after 4× compression.These reconstruction differences directly predict downstream task performance.
  • Downstream impact: Summarization and clone detection can remain competitive despite high Token Error prevalence because they rely on high-level semantic patterns rather than character-level precision.Thus, imperfect reconstruction does not uniformly determine downstream task performance.
  • Error hierarchy: Token Errors emerge first at 1×–2×, Line Errors at 2×–4×, and Block Errors at 4×–8× compression for most models.Token mistakes include confusing visually similar characters or missing punctuation; aggressive compression can produce hallucinated code in weaker models.
  • Error hierarchy: Gemini-3 maintains low block error rates even at 8× compression, explaining its stable downstream performance across compression levels.Other models show a rapid increase in Line and Block Errors at aggressive compression.

5 Discussion

The discussion finds that visual code processing adds minimal latency overhead, so demonstrated compression ratios can translate into comparable inference speedups. It also addresses internal and external validity through contamination controls, repeated statistical testing, cross-language replication, and evaluation across seven diverse MLLMs.

  • Efficiency: Latency curves for image and text inputs are comparable at identical token scales, indicating minimal overhead from visual encoding.The study benchmarks locally because API latency depends heavily on network conditions and server load.
  • Efficiency: 4× compressed images are approximately 4× faster than raw text when per-token latency parity holds.The 2×–4× compression ratios from RQ2–3 therefore translate directly into equivalent inference speedups.
  • Internal Validity: Internal validity is addressed through No Context baselines, a Git-derived CodeQA dataset, unanimous validation by three researchers, and five repeated experiments.Wilcoxon signed-rank tests assess statistical significance, while samples with annotation disagreement are discarded.
  • External Validity: Key findings are replicated on Java and evaluated across seven MLLMs spanning open-weight and proprietary model families.The model set includes Qwen-3-VL, GLM-4.6v, GPT-5-mini, GPT-5.1, Gemini-2.5-Pro, Gemini-3-Flash, and Gemini-3-Pro.

6 CodeOCR: Code Transformation Tool

CodeOCR is a practical middleware that renders source code as configurable images for MLLM-based code understanding. It combines syntax-highlighted visual rendering with dynamic resolution compression guided by user-specified token budgets.

  • Workflow: CodeOCR renders source code into compact images and passes them to an MLLM with user instructions, returning the model’s output.Users provide code and instructions as input; the rendered image and instructions are sent to the MLLM.
  • Workflow: The transformation has two stages: Visual Rendering creates syntax-highlighted images, while Dynamic Compression adjusts resolution to target specified compression ratios.Dynamic Compression operates according to user-specified token budgets.
  • Implementation: CodeOCR uses Pygments for syntax analysis and Pillow for image rendering and processing.These libraries support the tool’s visual transformation pipeline.

7 Related Work

Prior code LLMs achieve strong performance across diverse tasks but process code as linear token sequences, creating scalability challenges as context length grows. This work positions image-based code representation as a complementary alternative to text compression, enabling continuous compression through resolution scaling.

  • Large Language Models for Code: Code LLMs including Codex, Code Llama, StarCoder, DeepSeek-Coder, and Qwen2.5-Coder achieve strong performance across diverse code tasks.The cited tasks include code generation and repair.
  • Large Language Models for Code: Linear token-sequence processing creates scalability challenges as code context length grows.This limitation motivates more efficient code representations.
  • Text and Image-Based Compression: Text-based compression selectively retains or drops tokens, causing information loss and preventing further compression of retained key tokens.The work explores image representation as a complementary paradigm using continuous compression through resolution scaling.

8 Conclusion and Future Directions

The paper presents the first comprehensive empirical study of visual code representation for code understanding, evaluating state-of-the-art MLLMs across four representative tasks. Its findings indicate that the paradigm is viable and practically beneficial, with image compression achieving competitive or superior performance.

  • Conclusion and Future Directions: The paper provides the first comprehensive empirical study of visual code representation as a new paradigm for code understanding.The study evaluates this paradigm systematically rather than examining it through an isolated experiment.
  • Conclusion and Future Directions: The evaluation covers four representative code-understanding tasks using state-of-the-art MLLMs.This broad evaluation supplies empirical evidence across multiple task settings.
  • Conclusion and Future Directions: Image compression can achieve competitive or even superior performance for visual code representation.The passage identifies compression as a central source of the paradigm’s practical benefit, though it does not provide a specific compression figure here.
Loading 2602.01785v3…