Source-linked AI summary
Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs
Sai Srinivas Kancheti, Aditya Sanjiv Kanade, Vineeth N. Balasubramanian, Tanuja Ganu
TL;DR
The paper asks whether text-centric Chain-of-Thought reasoning transfers from mathematical and logical tasks to generalized visual spatial intelligence. It evaluates seventeen models across thirteen spatial benchmarks and uses No-Image++ ablations to test grounding. CoT consistently degrades spatial performance, while reasoning models hallucinate visual details from textual priors without images, motivating vision-centric reasoning paradigms.
Problem
Whether text-centric Chain-of-Thought reasoning transfers to generalized visual spatial intelligence remains insufficiently established, despite progress in mathematical and logical reasoning.
Method
The study evaluates seventeen models across thirteen spatial benchmarks, compares CoT and non-CoT prompting, and applies No-Image++ ablations with blank images and a Cannot determine option.
Results
CoT prompting consistently degrades visual spatial reasoning, while No-Image++ shows reasoning models hallucinate visual details and select answers from textual priors without image information.
Takeaways & Limitations
The findings indicate that scaling text-based reasoning alone is insufficient for robust spatial intelligence and highlight vision-centric training paradigms.
Takeaways & Limitations
The thirteen datasets do not represent the entirety of visual spatial reasoning, confounding factors remain difficult to isolate, and proprietary training details are opaque.
Abstract
from arXiv · showhide
Multimodal Reasoning Models (MRMs) leveraging Chain-of-Thought (CoT) based thinking have revolutionized mathematical and logical problem-solving. However, we show that this paradigm struggles with generalized spatial intelligence. We perform a comprehensive evaluation of seventeen models across thirteen spatial benchmarks and identify a critical gap: CoT prompting consistently degrades performance in visual spatial reasoning. Furthermore, through a novel No-Image++ ablation, we demonstrate that MRMs and CoT prompted MLMs suffer from severe shortcut learning, and hallucinate visual details from textual priors even when the image is absent. These findings challenge the efficacy of text-only CoT for spatial tasks and underscore the need for vision-centric reasoning paradigms.
1 Introduction
Multimodal reasoning models have advanced mathematical and logical problem-solving, but whether text-centric Chain-of-Thought reasoning transfers to spatial intelligence remains unresolved. This study evaluates that question and reports degraded spatial performance, hallucinated visual details, and a need for vision-centric reasoning.
- MRMs use supervised fine-tuning, reinforcement learning, and long Chain-of-Thought inference to generate step-by-step reasoning and self-correct on complex problems.
- The study evaluates seventeen models across thirteen datasets spanning static 2D relations, 3D geometry, and dynamic or temporal understanding.
- CoT prompting degrades visual spatial performance across diverse model backbones and scales, including a model explicitly enhanced for spatial perception.
- No-Image++ results show reasoning models hallucinate visual details and confidently select incorrect answers from textual priors despite blank images and a Cannot determine option.
- The findings challenge text-only CoT for spatial tasks and highlight vision-centric training paradigms as a needed direction.
2 Methodology
The methodology benchmarks diverse multimodal reasoning and language models on broad spatial datasets using standardized multiple-choice evaluation and pass@1 accuracy. It also includes No-Image ablations and controlled generation scoring to examine shortcut behavior and evaluation reliability.
- Baselines: The benchmark covers seventeen models, including Qwen2.5-VL backbones, eight RLVR-trained MRMs, additional InternVL and LLaVA models, Qwen3-VL-8B-Thinking, and GPT-4o.
- CoT comparison: The study compares native CoT prompts with base non-CoT prompts to isolate the effect of reasoning on spatial accuracy.
- Datasets: The thirteen datasets span static 2D relations, 3D geometry, depth, multi-image consistency, and temporal reasoning in natural scenes.
- Evaluation: All benchmarks use multiple-choice questions with uniform prompts and formats, while OmniSpatial and MindCube receive dataset-specific prompt additions.
- Evaluation: Performance is measured with pass@1 accuracy under greedy decoding across three seeds, using vLLM inference in bfloat16 precision.
- Scoring: An LLM judge scores generations, and rescoring with GPT-4o yields Cohen’s kappa greater than 0.99, indicating near-perfect agreement.
- Ablations: No-Image ablations replace images with full-gray inputs, with No-Image++ additionally offering a Cannot determine option to test question-only shortcuts.
3 Results and Analysis
Across visual spatial benchmarks, Chain-of-Thought prompting frequently harms performance, while reasoning models can rely on textual priors and hallucinate visual content. This pattern extends to proprietary models, where concise traces may reduce degradation.
- Six of eight open-source MRMs achieve higher accuracy with non-CoT prompts than with their native CoT prompts.GThinker shows a −23.14% performance drop, while Qwen3-VL-8B-Thinking shows a +0.64% non-CoT advantage at an approximately 65% baseline.
- Seven of eight open-source MRMs fail to surpass the non-CoT Qwen2.5-VL backbone’s 62.68% average accuracy across 13 datasets.ViGoRL and TreeVGR underperform by −2% and −1.57%, respectively; Vision-G1 is the only exception at +0.6%.
- No-Image evaluation shows that MRMs can answer spatial questions without informative visual input, with GThinker achieving 44.17% across 13 datasets.The analysis attributes this shortcut behavior to reliance on question text, options, and world-knowledge priors.
- GThinker produces degenerate repeated tool-call output under a non-CoT prompt, illustrating sensitivity to prompt-format changes.The qualitative example shows repetition continuing until termination.
- Non-CoT performance remains competitive with or exceeds CoT performance across the evaluated proprietary models.GPT-5 and GPT-5-nano show non-CoT advantages of +0.65% and +1.23%, while GPT-4o and GPT-4.1-mini show gains below 0.5%.
- Proprietary models generate substantially shorter reasoning traces than open-source models and lack common reflective or repetitive patterns.GPT-5-mini produces approximately 350 characters versus approximately 3600 for Qwen3-VL-8B-Thinking.
4 Conclusion
The paper finds that text-based Chain-of-Thought reasoning does not yet transfer from mathematical and logical success to visual spatial intelligence. Its No-Image++ analysis links the failure to hallucinated visual information based on textual priors.
- Benchmarking 17 models across 13 datasets finds consistent spatial degradation with CoT and frequent underperformance by specialized MRMs relative to their base models.The No-Image++ analysis identifies hallucinated visual information from textual priors as the mechanism behind this failure.
- The paper highlights vision-centric training paradigms, including visual verifiers and visual process reward models, as future directions.The proposed verifiers would check reasoning steps against image evidence, while process rewards would encourage grounded perception-first reasoning.
Limitations
The study covers a broad but non-exhaustive selection of visual spatial datasets and R1-style MRMs, with confounding factors difficult to isolate and proprietary training details opaque.
- The 13 included datasets do not represent the entirety of the visual spatial reasoning domain.The authors also note that completely isolating factors behind performance changes is challenging.
- Opaque proprietary-model training details limit deeper analysis of proprietary model behavior.
A Prompts
The appendix specifies the prompts used for baseline systems, spatial datasets, and LLM-based scoring. It distinguishes a simple no-thinking base prompt from model-specific CoT instructions and deterministic answer-matching rules.
- A Prompts: The appendix introduces the dataset and system prompts used in the baseline evaluations.
- A.1 System Prompts: The base prompt asks Qwen2.5-VL-7B to solve spatial questions as a spatial-reasoning assistant without explicit thinking instructions.
- A.1 System Prompts: CoT prompts instruct reasoning models to generate step-by-step internal reasoning before their final answers, commonly using <think> and <answer> tags.
- A.1 System Prompts: ViGoRL-Spatial’s prompt requires checking possible solutions and image regions while grounding reasoning in objects, relationships, and (x,y) coordinates.
- A.2 Dataset Prompts: Dataset-specific instructions define image-based spatial queries for OmniSpatial and multi-view scene analysis for MindCube.
- A.3 LLM Judge Scoring Prompts: MCQ scoring maps a free-form response to an option letter, returning 0 when no option clearly matches and never judging factual correctness.
- A.3 LLM Judge Scoring Prompts: VQA scoring returns 1 only when the response expresses the same final answer as the gold answer, ignoring reasoning steps and minor surface differences.
B Expanded Tables
The expanded materials describe the benchmark coverage and methodological details of several reasoning-model baselines. They include spatial capabilities spanning 2D relations, depth, orientation, localization, scale, counting, 3D geometry, motion, and related settings.
- Baseline methods: GThinker uses visual-clue tags, reflection-oriented rethinking, a 7k-sample CoT dataset, cascaded MLM generation, and diverse RL sources.The passage also mentions 4K samples selected from RL sources.
- Benchmark coverage: The benchmark taxonomy covers object relations, depth, orientation, localization, scale comparison, counting, 3D geometry, multi-image reasoning, motion, reference frames, interaction, and attributes.The dataset tags distinguish these spatial capabilities, while starred sizes indicate circular evaluation.
- Baseline methods: Vision-R1 uses two-stage CoT SFT and GRPO training with Progressive Thinking Suppression Training on diverse mathematical and spatial datasets.The reported training setup includes 200k cold-start samples and 10k RL samples.
C Qualitative Examples
The paper provides additional qualitative examples to illustrate the failure modes identified in its analysis.
- Qualitative examples: Additional qualitative examples illustrate the failure modes identified in the analysis.The passage introduces these examples without specifying their individual cases.
C.1 GThinker Degenerate Output
Under a non-CoT direct-answer prompt, GThinker can produce degenerate repeated outputs instead of answering normally. This behavior explains a reported 23.14% performance drop.
- Failure case: GThinker fails to follow the non-CoT direct-answer format and produces repeated degenerate outputs until the maximum token limit.A RealWorldQA example shows repeated tool-call markers for a simple comparison question.
- Reported impact: −23.14% is the reported GThinker performance drop associated with this degenerate behavior.The paper links the drop to the ill-formed output behavior shown in the example.
C.2 No-Image++ Hallucination
In the No-Image++ setting, models receive a blank image yet still generate detailed spatial explanations and select answers. The example attributes this behavior to hallucinated visual details and textual priors.
- No-Image++ setup: The No-Image++ ablation replaces the input image with a blank gray image and adds a “Cannot determine from the image” option.The setting tests whether models acknowledge the absence of visual information.
- Hallucinated reasoning: ViGoRL answers “below” for the cave’s position relative to trees despite receiving no visual information.Its reasoning invents image coordinates and describes the cave as lower than the trees.
- Shortcut behavior: The model hallucinates spatial coordinates and relies on world knowledge, such as caves typically being below trees, instead of acknowledging missing visual evidence.This example connects the answer to textual or conceptual priors rather than image-grounded perception.
- Dataset-wise results: Qwen3-VL-8B-Thinking outperforms CoT on 8 of 13 datasets under the reported dataset-wise comparison, with a +0.64% average difference.This result is reported in Table 9 for CoT versus non-CoT performance.