PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do

Zhuoran Jin, Kejian Zhu, Hongbang Yuan, Yupu Hao, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

arXiv:2606.22565v1cs.CLcs.AIcs.CV

TL;DR

The effectiveness of multimodal Chain-of-Thought reasoning across perception and reasoning tasks remains unclear. Evaluating it across 12 tasks, the paper finds that CoT is task-dependent, open-source reasoning models yield marginal gains, and visual reflection declines during reasoning.

  • Problem

    Whether multimodal CoT consistently improves performance across diverse perception and reasoning tasks remains an open question.

  • Method

    The paper compares direct answering with CoT reasoning across 12 multimodal tasks and evaluates non-reasoning models alongside reasoning variants.

  • Results

    CoT improves mathematical, scientific, and multi-image reasoning but can harm perception; reasoning models gain marginally, while visual reflection declines during reasoning.

  • Takeaways & Limitations

    Multimodal CoT should be applied selectively, while improving visual reflection is a promising direction for addressing its bottleneck.

  • Takeaways & Limitations

    The evaluation covers only a subset of datasets and models across 12 tasks, so it may not capture the full diversity of multimodal tasks.

Abstract

from arXiv · show

Chain-of-Thought (CoT) has become a standard method for improving reasoning capabilities in large language models (LLMs) by eliciting step-by-step thinking, but its effectiveness in multimodal tasks remains unclear. In this paper, we aim to systematically investigate the key question: What can multimodal Chain-of-Thought reasoning do, and where and why does it fall short? To this end, we evaluate 12 multimodal tasks across perception and reasoning categories using both 14 non-reasoning models and 8 reasoning models. Our analysis reveals several important findings: (1) CoT is not a free lunch and should be used selectively depending on the specific requirements of each task. For perception tasks, CoT can lead to undesirable side effects, such as reduced performance in visual grounding and object counting. In contrast, it proves effective for reasoning tasks involving mathematical, scientific, and multi-image reasoning; (2) Compared to original models, existing open-source multimodal reasoning models often yield only marginal overall improvements, possibly due to an overemphasis on mathematical reasoning at the expense of broader capabilities; (3) Visual reasoning remains a key bottleneck for current multimodal CoT, as models exhibit a Look Light, Think Heavy pattern where verbal reflection rises and falls during reasoning, whereas visual reflection consistently diminishes. These findings suggest that while multimodal CoT handles verbal reflection relatively well, it lacks the ability to maintain deep visual introspection throughout the reasoning process.

1 Introduction

The paper systematically examines multimodal Chain-of-Thought across perception and reasoning tasks, finding that its benefits are task-dependent. It identifies visual reasoning as a primary bottleneck behind a “Look Light, Think Heavy” pattern in current multimodal reasoning models.

  • RQ1: The study evaluates multimodal tasks across perception and reasoning dimensions, including OCR, visual grounding, hallucination, object counting, mathematical, scientific, logical, algorithmic, and spatial reasoning.The perception–reasoning categorization provides the paper’s analytical framework.
  • RQ1: Multimodal CoT is not a free lunch: it can harm visual grounding, knowledge-based VQA, and object counting, so it should be used selectively.The study compares direct answering with CoT across 12 multimodal perception and reasoning tasks.
  • RQ2: Existing multimodal reasoning models are compared with non-reasoning models to test whether reinforcement-learning-based test-time scaling transfers effectively from LLMs to MLLMs.This comparison addresses whether longer multimodal CoT reliably improves performance over base models.
  • RQ3: Visual reasoning is a primary bottleneck limiting multimodal CoT effectiveness and overall performance.The paper investigates this limitation using visual and textual reasoning probes across multimodal reasoning tasks.
  • RQ3: Current models exhibit “Look Light, Think Heavy”: verbal reflection rises and falls during reasoning, while visual reflection steadily declines.Extended reasoning also shifts disproportionate attention toward reasoning tokens and progressively neglects visual tokens.

2 Problem Formulation

The paper formulates multimodal CoT as generating a long reasoning sequence and final answer from image inputs, a textual question, and a prompting prefix. It evaluates CoT across perception and reasoning tasks using both non-reasoning and reasoning models.

  • 2.1 Multimodal Chain-of-Thought: Multimodal CoT maps image inputs I, textual question q, and prompting prefix p_c to a reasoning sequence r and final answer a.The reasoning sequence captures step-by-step reasoning leading to the answer, whereas direct answering produces a shorter output.
  • 2.2 Perception and Reasoning Tasks: The evaluation categorizes multimodal tasks into perception and reasoning dimensions.This framing is used to holistically assess CoT’s impact across different multimodal capabilities.
  • 2.2 Perception and Reasoning Tasks: Perception covers comprehensive evaluation, OCR, visual grounding, hallucination detection, knowledge-based VQA, and object counting.These tasks focus on fine-grained visual understanding and cross-modal alignment.
  • 2.2 Perception and Reasoning Tasks: Reasoning covers mathematical, scientific, logical, algorithmic, spatial, and multi-image reasoning.These tasks emphasize multi-step reasoning grounded in visual and textual inputs.
  • 2.3 Evaluation Models: The experiments compare non-reasoning models under direct answering versus CoT prompting.Non-reasoning models are treated as general models in the evaluation.
  • 2.3 Evaluation Models: Reasoning models generate much longer CoT sequences and exhibit reflection that enables self-correction.For reasoning models with test-time scaling, performance is analyzed against corresponding non-reasoning models.

3 Strengths and Pitfalls of Multimodal Chain-of-Thought

Multimodal CoT is task-dependent: it can harm perception while improving several reasoning tasks, with scale affecting logical and algorithmic reasoning. Open-source multimodal reasoning models generally deliver modest gains, potentially because math-focused training neglects broader capabilities.

  • Perception tasks: CoT causes average performance drops of 4.6%, 3.3%, and 4.8% on visual grounding, knowledge-based VQA, and object counting, respectively.These results show that CoT can introduce negative side effects in perception tasks.
  • Reasoning tasks: CoT improves mathematical, scientific, and multi-image reasoning by 6.1%, 2.9%, and 4.9%, respectively.Mathematical and scientific reasoning primarily depend on text-dominant reasoning after basic visual understanding.
  • Reasoning tasks: For logical and algorithmic reasoning, larger models often benefit from CoT, whereas smaller models show limited, degraded, or negative gains.CoT effectiveness in these tasks is closely related to model scale because they rely more heavily on visual-information reasoning.
  • Reasoning models: Open-source multimodal reasoning models often show only limited or modest gains across diverse tasks.The paper suggests that predominant RLVR training on math-related questions may overemphasize mathematical reasoning while neglecting broader reasoning abilities.
  • Reasoning models: Gemini-2.0-Flash-Thinking demonstrates substantial and consistent gains across diverse reasoning tasks, unlike the limited gains often observed in open-source models.The comparison covers original and reasoning-enhanced versions of five open-source models and one commercial model.

4 Shallow Visual Reflection in Multimodal Chain-of-Thought

Visual reasoning is a central bottleneck in multimodal CoT: models perform worse on visual probes, and visual reasoning correlates more strongly with overall task performance than textual reasoning. Their reflection is also shallow, with visual attention and introspection declining as verbal reasoning develops.

  • Visual reasoning bottleneck: Over 80% of errors in logical reasoning tasks arise from incorrect reasoning over visual information.The analysis examines visual and textual contributions using intermediate probe tasks within multimodal reasoning problems.
  • Visual reasoning bottleneck: 20% performance drop on visual reasoning probes occurs relative to textual reasoning probes, while visual probe accuracy correlates more strongly with overall task performance.These results identify visual reasoning as a key bottleneck for multimodal large language models.
  • Reflection dynamics: Existing multimodal reasoning models exhibit a “Look Light, Think Heavy” pattern: verbal reflection rises and falls, while visual reflection steadily declines throughout CoT.Reasoning models show more visual and verbal reflection than non-reasoning models, but the two behaviours follow opposite trends.
  • Reflection dynamics: When critical visual cues are occluded, models increase both visual and verbal reflection but show limited ability to abstain from answering.This reveals shallow visual reflection despite explicit reflective behaviour under missing-information conditions.
  • Internal attention mechanism: During long CoT, multimodal reasoning models increasingly prioritize reasoning tokens while gradually neglecting visual inputs.This attention imbalance shifts focus away from the original visual input and may constrain visual reflection.

5 Relate Works

Prior work shows that Chain-of-Thought prompting improves mathematical and coding performance through intermediate reasoning, while newer approaches extend it with reflection and multimodal reasoning. Recent studies also report that CoT gains vary substantially across tasks and expose limitations in current paradigms.

6 Conclusion

The study concludes that multimodal CoT should be applied selectively because its effectiveness depends on the task. It also identifies marginal gains in current open-source models and diminishing visual reflection as central limitations.

  • Key findings: CoT’s efficacy is task-dependent and requires selective application.The conclusion emphasizes that multimodal CoT has both strengths and limitations rather than universally improving performance.
  • Key findings: Current open-source models show only marginal gains, likely because of overemphasis on mathematical reasoning.The paper attributes this possible limitation to prioritizing mathematical reasoning over broader multimodal capabilities.
  • Key findings: Visual reasoning remains a bottleneck, with visual reflection diminishing relative to verbal reflection in a “Look Light, Think Heavy” pattern.The conclusion characterizes this pattern as a disparity between declining visual reflection and comparatively stronger verbal reflection.

Limitations · A Task Details · A.1 Multimodal Perception Tasks

The study evaluates a broad but computationally constrained subset of multimodal tasks and proposes visual reflection as a direction for handling ambiguous visual inputs. The task details specify benchmark-based evaluations spanning comprehensive assessment, grounding, hallucination, knowledge-based VQA, and object counting.

  • Limitations: The evaluation covers 1–3 datasets per task across 12 multimodal tasks, using 14 general models and 8 reasoning models, but may not capture task diversity.The authors attribute this limitation to computational constraints and plan to expand the datasets evaluated.
  • Limitations: Visual reflection is proposed as a future direction, with o3 recognizing obscured information, inspecting the occluded region, and appropriately refusing when visual input is insufficient.The example involves mosaicked images and explicitly cropping, zooming, and revisiting critical visual regions.
  • A Task Details: Table 1 overviews the datasets and sample sizes used across the evaluated tasks, with detailed descriptions provided in the following sections.This appendix organizes the task-specific evaluation settings after the overview table.
  • A.1 Multimodal Perception Tasks: The comprehensive evaluation samples 200 questions each from MME, MMStar, and MMT-Bench to assess broad multimodal capabilities.MME targets multitask performance, MMStar addresses visual independence and data leakage, and MMT-Bench focuses on multimodal benchmarking.
  • A.1 Multimodal Perception Tasks: Visual grounding evaluates cross-modal alignment by localizing textual referents with predicted bounding boxes, using 150 instances each from RefCOCO and RefCOCOg.The task measures recognition and localization of visual entities.
  • A.1 Multimodal Perception Tasks: Multimodal hallucination evaluation measures whether outputs remain grounded in visual and other input evidence, sampling 250 tasks each from HallusionBench and POPE.The evaluation targets factual consistency between generated responses and multimodal evidence.
  • A.1 Multimodal Perception Tasks: Knowledge-based VQA tests commonsense and world knowledge beyond directly observable image content, using 200 A-OKVQA questions.The appendix contrasts this task with standard VQA and illustrates Gemini-2.0-Flash responses.
  • A.1 Multimodal Perception Tasks: Object counting assesses visual perception by asking models to count distinct entities, using 200 Super-CLEVR samples that include realistic object types.Super-CLEVR extends CLEVR beyond simple geometric shapes, and the appendix compares Qwen2.5-VL-7B-Instruct responses.

A.2 Multimodal Reasoning Tasks

The multimodal reasoning evaluation covers mathematical, scientific, logical, algorithmic, and spatial reasoning, using 200 sampled tasks from a designated benchmark or dataset for each category. Figures 16–20 illustrate direct-versus-CoT responses for these tasks.

  • Mathematical Reasoning: Mathematical reasoning uses 200 MathVerse tasks involving mathematical concepts, multi-step inference, and precise computation.Figure 16 compares direct and CoT responses generated by Qwen2.5-VL-72B-Instruct.
  • Scientific Reasoning: Scientific reasoning evaluates multimodal scientific question answering with 200 graduate-level MMMU tasks spanning text, charts, images, and diverse disciplines.Figure 17 compares direct and CoT responses generated by InternVL3-38B.
  • Logical Reasoning: Logical reasoning uses 200 PuzzleVQA visual puzzles requiring multimodal conclusions, predictions, pattern recognition, and problem solving.Figure 18 compares direct and CoT responses generated by GPT-4o.
  • Algorithmic Reasoning: Algorithmic reasoning uses 200 AlgoPuzzleVQA tasks covering step-by-step computational procedures such as graph theory, combinatorics, and search problems.Figure 19 compares direct and CoT responses generated by Claude-3-7-Sonnet-Thinking.
  • Spatial Reasoning: Spatial reasoning evaluates multimodal analysis of object position, orientation, distance, and movement using 200 SpatialEval tasks.Figure 20 compares direct and CoT responses generated by Qwen2.5-VL-72B-Instruct.

B Evaluation Details · C Prompts for Textual and Visual Reasoning Probe

The evaluation covers 14 non-reasoning and 8 reasoning multimodal models with task-dependent prompting. Textual and visual reasoning probes are automatically generated and filtered, then manually verified on 400 samples for reliability.

  • B Evaluation Details: Prompts vary by task: models with specific prompts retain their original designs, while others use a standardized prompt.The evaluation includes prompts for comprehensive evaluation, OCR, visual grounding, hallucination, knowledge-based VQA, object counting, and multiple reasoning tasks.
  • C Prompts for Textual and Visual Reasoning Probe: o4-mini generates visual and textual reasoning probes, and GPT-4.1 filters them.The automatic generation and filtering process may introduce minor errors.
  • C Prompts for Textual and Visual Reasoning Probe: 400 probe samples are manually verified to ensure accuracy and reliability, producing probes with high correctness.The probes use separate generation and judgment prompts for textual and visual reasoning, as documented in Tables 14–17.
  • C Prompts for Textual and Visual Reasoning Probe: Verbal and visual reflection are annotated step by step to characterize multimodal CoT reasoning.The annotation prompt is provided in Table 18, and the resulting reflection distributions are summarized in Figure 22.

D Implementation Details

The implementation uses a fixed inference setup across experiments and manually classifies multimodal CoT failures by error type. The reported taxonomy includes visual reasoning, textual reasoning, and other errors.

  • Experimental setup: All experiments use vllm for open-source MLLM inference on 4×A100 80GB GPUs with temperature set to 0.7.These settings are applied across all models.
  • Error analysis: Failure cases are manually classified into categories including visual reasoning errors, where perception is correct but deductions from visual evidence are incorrect.This category targets reasoning failures after successful visual perception.
  • Error analysis: The taxonomy also includes textual reasoning errors involving noncompliant answer formats and other errors that do not fit the listed categories.Examples of textual reasoning errors include ambiguous responses.

E Additional Experimental Results · F Case Study of o3

Additional experiments examine correlations between task performance and reasoning-probe accuracy and visualize model attention during mathematical and logical reasoning. The o3 case studies illustrate uncertainty in visual interpretation, refusal when key image information is missing, and use of external tools for localization and algorithm execution.

  • E Additional Experimental Results: Correlation analysis compares overall task performance with logical-task reasoning-probe accuracy for visual and textual reasoning across models.The figure reports Pearson correlation coefficients r for the visual and textual reasoning probes.
  • E Additional Experimental Results: Attention visualizations examine Kimi-VL-A3B-Thinking on mathematical reasoning.
  • E Additional Experimental Results: Attention visualizations also examine Kimi-VL-A3B-Thinking on logical reasoning.
  • E Additional Experimental Results: Additional attention visualizations cover Qwen3-Omni-30B-A3B-Thinking and Qwen3-VL-8B-Thinking on mathematical reasoning.
  • E Additional Experimental Results: Qwen3-VL-30B-A3B-Thinking is likewise analyzed through attention visualizations on mathematical reasoning.
  • F Case Study of o3: In an o3 geometry case study, the model interprets the image as a possible pentagon and speculates that a blue bar may obscure a vertex label before considering the angles.The response identifies visible labels E, F, H, and J and hypothesizes a missing G.
  • F Case Study of o3: The o3 case studies include refusing to answer when images lack key information and leveraging external tools for visual localization and algorithm execution.
  • F Case Study of o3: Another o3 case study concerns completing a 9 * 9 chessboard queen-placement problem with seven queens already placed.
Loading 2606.22565v1…