PaperlayerView exact revision on arXiv

Source-linked AI summary

Optical Reasoning: Rethinking Images as an Expressive Reasoning Medium Beyond Text

Yutong Bian, Dongjie Cheng, Heming Xia, Yongqi Li, Wenjie Li

arXiv:2606.09585v1cs.AI

TL;DR

Existing multimodal reasoning uses textual or interleaved textual-visual rationales, motivating the question of whether images alone can support reasoning. The paper proposes optical reasoning with typographic and graphical variants, finding that it can match or exceed text reasoning while using fewer tokens and reaching 1.96× text-reasoning token efficiency.

  • Problem

    The paper asks whether images alone can serve as a reasoning medium for both language and multimodal tasks.

  • Method

    Optical reasoning represents intermediate rationales as images, using typographic layouts for compact rendering and graphical compositions for text, formulas, visual structures, and spatial layouts.

  • Results

    Optical reasoning matches or exceeds text reasoning while reducing reasoning tokens by 28.57% on language tasks and 16% on multimodal tasks.

  • Takeaways & Limitations

    Images achieve 1.96× the token efficiency of text reasoning and provide a unified canvas for structuring interleaved-modal rationales.

  • Takeaways & Limitations

    Effectiveness depends on model-specific visual sensitivities, and graphical rationales may contain inaccuracies.

Abstract

from arXiv · show

Chain-of-Thought (CoT) improves the performance of Large Language Models (LLMs) and has been extended to Multimodal Large Language Models (MLLMs). More recent work further moves from text-based multimodal reasoning toward interleaved-modal reasoning, where intermediate steps can incorporate both textual rationales and visual evidence. In this work, we propose a bolder and more ambitious idea: could images alone serve as the reasoning medium for both language and multimodal tasks? To explore this, we propose optical reasoning, which treats images as a standalone reasoning medium. We instantiate this concept with two variants: typographic-based optical reasoning, which optimizes visual layouts for compact rationale rendering, and graphical-based optical reasoning, which composes text and graphical elements into structured visual rationales. Across mathematical, scientific, and interleaved-modal reasoning benchmarks, optical reasoning can match or even exceed traditional text reasoning while reducing reasoning tokens by an average of 28.57% on language tasks and 16% on multimodal tasks, achieving 1.96 times the token efficiency of text reasoning. These results show that images can effectively and efficiently encode rationales while providing a unified visual canvas for reasoning.

1 Introduction

The paper proposes optical reasoning, which uses images alone to represent rationales for language and multimodal tasks. It develops typographic and graphical variants and reports competitive reasoning performance with reduced token use.

  • Motivation: Optical reasoning asks whether images alone can serve as the reasoning medium for language and multimodal tasks.The motivation is to compress rationales while integrating textual, graphical, and spatial information on a unified visual canvas.
  • Method: The approach treats images as the sole reasoning medium through typographic-based and graphical-based variants.T-OR optimizes text width and font size for dense, readable layouts; G-OR assigns rationale steps to visual panels containing text, formulas, graphical structures, and spatial relations.
  • Results: 28.57% fewer reasoning tokens on language tasks and 16% fewer on multimodal tasks accompany performance that matches or exceeds text reasoning in many model-benchmark pairs.When T-OR underperforms, the reported average accuracy gaps are 0.027 for language tasks and 0.014 for multimodal tasks, with token reductions of 20% and 32%, respectively.
  • Results: 1.96× token efficiency relative to text reasoning is reported across the evaluated reasoning settings.The study evaluates mathematical, scientific, and interleaved-modal benchmarks with five frontier MLLMs.
  • Evaluation: The paper evaluates optical reasoning across five benchmarks and five advanced MLLMs.The evaluation spans mathematical reasoning, scientific reasoning, and interleaved-modal reasoning.

2 Related Work

Related work extends reasoning from text-based CoT toward interleaved-modal reasoning and explores images as a mechanism for compressing textual contexts and rationales.

  • Interleaved-modal reasoning: Interleaved-modal reasoning incorporates visual regions, visual tokens, or external perception tools alongside textual reasoning steps.Examples include ICoT, MINT-CoT, and DeepEyes.
  • Optical compression: Optical compression encodes textual contexts or prior rationales as images to reduce the token burden of visual-language processing.The cited line of work includes DeepSeek-OCR, Glyph, CodeOCR, AgentOCR, MemOCR, VTC-R1, and RoT.

3 Method

This section formalizes optical reasoning as image-based rationale representation, replacing text rationale tokens with visual reasoning tokens. It instantiates the framework with typographic and graphical renderers that optimize compactness or organize multimodal content spatially.

  • 3.1 Preliminary: Text Reasoning: Text reasoning serializes textual rationale units, such as spans or equations, into text tokens before answer decoding.The rationale sequence is assumed to be generated by a model or provided externally.
  • 3.2 Optical Reasoning: Optical reasoning represents intermediate rationales as images and uses image tokens, rather than text tokens, as the reasoning medium.The model derives answers from question tokens and visual reasoning tokens produced by encoding the rationale image.
  • Typographic-based optical reasoning: T-OR renders interleaved rationales into compact typographic images while preserving the original order of textual and visual units.Textual units can be typeset as text, equations, or tables, while visual units are inserted as image blocks.
  • Typographic-based optical reasoning: T-OR searches candidate widths and font sizes for a feasible layout under budget B, selecting a compact, readable configuration using fill ratio and layout penalties.The score rewards canvas utilization while penalizing excessive margins and overly tight layouts that reduce readability.
  • Graphical-based optical reasoning: G-OR decomposes rationales into reasoning steps and assigns each step to a visual panel containing graphical structures alongside retained text and equations.This step-aligned composition reorganizes the rationale within a unified visual canvas.

4 Experiments

Experiments evaluate optical reasoning across mathematical, scientific, and interleaved-modal benchmarks using five frontier MLLMs, comparing no reasoning, text reasoning, and visual rationale settings. T-OR generally preserves or improves accuracy while reducing reasoning tokens, and additional studies examine graphical reasoning, rendering choices, layout factors, compression, and alternative rationale-compression baselines.

  • Experimental Setup: The evaluation covers mathematical, scientific, and interleaved-modal reasoning with five frontier MLLMs, using accuracy as the primary metric.The models include GPT-5.1, Gemini 2.5 Flash, Claude Sonnet 4.5, Kimi K2.5, and Qwen3-VL-235B.
  • Typographic-based Optical Reasoning: T-OR matches or outperforms text reasoning across multiple model-benchmark pairs while achieving an average MAG 1.96× higher than text reasoning across token-budget ratios.On ZebraCoT, T-OR outperforms text reasoning on Gemini 2.5 Flash, Kimi K2.5, and GPT-5.1, while remaining within 1% on the other two models.
  • Graphical-based Optical Reasoning: G-OR achieves the best AquaRat accuracy among no reasoning, text reasoning, and T-OR settings.Its composition integrates text, graphical elements, and spatial layouts within a unified visual canvas.
  • Ablation and In-depth Analysis: T-OR remains effective under extreme compression, achieving accuracy that can outperform the no-reasoning baseline at a token budget ratio of −98.75%.The evaluation on AquaRat with Gemini 2.5 Flash reduces the average estimated budget to as few as 1.2 tokens per example, with optimal performance at an average budget of 7.2 reasoning tokens.
  • Ablation and In-depth Analysis: Renderer and layout choices affect optical-reasoning accuracy: models prefer different rendering engines, while readable colors, fonts, sizes, and widths perform better.Qwen3-VL and Claude perform best with XeLaTeX, Gemini performs best with Matplotlib, red outperforms the black baseline, and Heros is the best font family in the reported ablations.
  • Ablation and In-depth Analysis: T-OR consistently outperforms LLMLingua-2 under equivalent reasoning tokens and the same rationales on AquaRat.The comparison attributes the difference to optical reasoning preserving rationale content rather than truncating text details.

5 Conclusion and Future Work

The paper positions optical reasoning as image-only reasoning, with typographic and graphical variants targeting compact encoding and unified visual organization. Across benchmarks, it reports effective and efficient reasoning while identifying model-dependent perception and graphical inaccuracies as limitations.

  • Optical reasoning treats images as the sole reasoning medium and instantiates this idea through typographic-based and graphical-based variants.The variants target compact rationale rendering and unified integration of text, graphical elements, and spatial layouts.
  • The typographic variant maximizes information density, while the graphical variant organizes rationales into step-aligned compositions.
  • Empirical results across diverse benchmarks and MLLMs show that images can serve as an effective and efficient reasoning medium.
  • Optical reasoning may vary across MLLMs because effectiveness depends on resolution, layout density, rendering style, and visual-token budgets.The paper suggests model-adaptive rendering strategies as a future direction for improving robustness.
  • Graphical rationales may contain graphical inaccuracies, motivating end-to-end fine-tuning or reinforcement learning with feedback on visual correctness and answer accuracy.

A Case Study

The case study shows that optical reasoning integrates textual, graphical, and spatial information on a unified visual canvas. Graphical compositions separate reasoning stages and expose intermediate relations, but their schematics can be inaccurate.

  • T-OR preserves original rationales while integrating text, graphical elements, and spatial layouts within a unified visual canvas.The physics example retains key equations and derivations.
  • G-OR separates reasoning stages through panels and diagrammatic decomposition, enabling multimodal illustrations of intermediate variables and spatial relations.
  • Images can support interleaved-modal CoT by combining textual rationales, graphical elements, and spatial relations in one visual canvas.
  • Graphical elements can express complex concepts vividly, but generated schematics are not always geometrically accurate.The geometric example contains a misplaced red segment relative to the intended diagonal constraint.
  • G-OR therefore introduces graphical hallucination as a potential failure mode alongside its stronger visual expressiveness.

B Prompt Templates

The appendix specifies prompt templates for baseline, typographic, and graphical reasoning conditions, along with answer evaluation procedures. These templates operationalize graphical rationales as compact, step-aligned visual canvases.

  • B Prompt Templates: The prompt-template section documents the evaluation prompts used in the study.
  • B.1 Evaluation Prompts: No-reasoning prompts contain only the problem, text reasoning adds a rationale, and free reasoning asks models to solve step by step.
  • B.1 Evaluation Prompts: T-OR prompts pair the problem text with a rendered typographic rationale image, while G-OR prompts pair it with a graphical rationale image.
  • B.2 Graphical Rationale Generation Prompt: G-OR generation converts an answer-masked rationale into a compact, step-aligned multi-panel visual canvas.
  • B.2 Graphical Rationale Generation Prompt: The G-OR prompt combines textual reasoning anchors with graphical elements and spatial layouts to carry the reasoning process.
  • B.3 LLM Judge Prompt: Answer evaluation first applies rule-based extraction and matching to model predictions.
  • B.3 LLM Judge Prompt: An LLM judge handles unresolved predictions by comparing only the final prediction with the gold answer and returning a structured correctness verdict.

C Patch Token Mapping

The paper estimates visual reasoning tokens through a uniform patch-based mapping applied across models. Rendered rationale images are resized to compare text and visual reasoning under matched or compressed budgets.

  • The evaluation applies a Qwen3-VL-style patch mapping uniformly across models to estimate visual reasoning tokens.This follows the setting of CodeOCR.
  • For a rendered reasoning image, the estimated visual token count is based on its height H and width W.
  • Because closed-source models do not disclose exact visual tokenization rules, the study uses the same patch-based estimator for comparable accounting.
  • T-OR images are resized to target token budgets corresponding to reduction ratios from −80% to 0% relative to text-reasoning token counts.
  • The resizing procedure enables controlled comparisons between text reasoning and visual reasoning under matched or compressed reasoning budgets.

D Benchmark Details

The benchmark details specify evaluation metrics, reproducible sampling for Zebra-CoT, and prompt templates for the main task settings and graphical rationale generation. Figures illustrate typographic and graphical optical reasoning across text-only and multimodal tasks.

  • The experiments report accuracy for task performance and MAG for token efficiency, following Eq. 14.
  • Zebra-CoT’s evaluation split contains 300 examples from 15 sampled subsets, constructed with reservoir sampling and random seed 42.
  • The benchmark documentation covers dataset categories, task formats, dataset sizes, and evaluation metrics used in the main evaluation.
  • G-OR generates a graphical rationale image as a separate visual input while requesting only the final answer in boxed format.
  • T-OR preserves rationales in dense typographic layouts, whereas G-OR reorganizes them into step-aligned graphical panels across text-only and multimodal tasks.
Loading 2606.09585v1…