When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought

1 Introduction

• Text-only CoT leaves a gap for visual problems involving spatial reasoning, geometric manipulation, or physical simulation.The introduction asks whether multimodal models can reason with integrated visual artifacts and how much those artifacts contribute to solving complex visual reasoning problems.Source: §1 · p. 2
• Existing multimodal benchmarks often test perception or keep multi-step reasoning textual, while tool-based visual approaches depend on the tools they use.Some benchmarks focus on pattern discovery or simple multiple-choice questions with limited visual clues.Source: §1 · p. 2
• MIRA supplies 546 examples with gold-standard intermediate visual states aligned to reasoning trajectories and final answers.Its 20 task types span single auxiliary images and sequences of visual states, from spatial layout and geometry to cross-temporal tracking.Source: §1 · p. 2
• No model exceeds 20% accuracy with direct inputs, while Visual-CoT improves GPT-5-mini from 13.7% to 23.2% on average.Text-CoT reduces accuracy for Gemini 2.5 Pro and o3 by 18.3% and 14.0%, respectively; Physics tasks nearly double across proprietary MLLMs with Visual-CoT.Source: §1 · p. 3

CoT Reasoning in LLMs

• Textual CoT methods [42] and variants [24] [55] add intermediate language reasoning, but remain text-only for inherently visual tasks.The paper describes diagrams and spatial representations as better suited to such tasks.Source: §2 · pp. 3–4

Reasoning Benchmarks in MLLMs

• VQA, captioning, and grounding benchmarks [16] [51] commonly use image-in, text-out evaluation, while multi-step datasets such as ScienceQA [27] keep reasoning hints textual.Broader suites [34] [51] [52] likewise do not require generating or manipulating intermediate visual cues; MIRA instead supplies human-annotated visual CoT.Source: §2 · p. 4

Integrating Visual Cues into Reasoning

• Visual CoT [36] adds visual cues, and tool-augmented systems [43], [19], and [39] generate auxiliary visuals, but these approaches rely on external tools and lack systematic evaluation in open-ended reasoning.Other frameworks and [22] generate or update diagrams with code for geometry and spatial reasoning.Source: §2 · p. 4

Unified MLLMs with Image Generation Capabilities

• Unified MLLMs combine image understanding and generation, but most are optimized for photorealistic synthesis or captions rather than task-specific diagrams.The paper contrasts these with understanding-focused models [3] [56] and notes that robust think-while-drawing ability has not been demonstrated by systems such as Gemini [12] and GPT-5 [23].Source: §2 · p. 4

3 MIRA: Multimodal Imagination for Reasoning Assessment

• MIRA contains 546 samples across four domains: Euclidean Geometry, Physics-Based Reasoning, Abstract Spatial & Logical Puzzles, and Causal Transformations.The benchmark uses human annotations and unique ground-truth answers, then analyzes multi-step visual reasoning with sequences of clues.Source: §3 · p. 4

3.1 Benchmark Data Design and Construction Details

• The data design requires intermediate visual information to solve questions, treating it as a complement to text-based CoT and verbal descriptions.Examples may require one pivotal visual clue or a sequence of visual trajectories, parameterizing complexity by reasoning steps and intermediate visuals.Source: §3.1 · p. 5
• A hybrid pipeline combines researcher-authored and programmatically generated problems, image refinement, and cross-review for unambiguous answers and reliable visual trajectories.The paper describes manual labeling, human inspection, and programmatic generation as components of construction.Source: §3.1 · pp. 4–5

3.2 Evaluation Protocol

• The three-level protocol compares direct image-and-question input, textual CoT, and supplied annotated visuals paired with textual reasoning prompts.Direct input measures overall problem-solving; Text-CoT tests text-based reasoning; Simulated Visual-CoT tests reasoning with provided intermediate images.Source: §3.2 · p. 5

4 Evaluation

• The evaluation measures MLLMs on visual-reasoning tasks, visual-cue gains, and capability limits; the setup compares direct, Text-CoT, and simulated Visual-CoT inputs.The evaluation uses micro-averaged accuracy and answer extraction via tags, regular expressions, and an LLM judge for remaining ambiguous outputs.Source: §4 · p. 6, §4.1 · p. 6

4.1 Experimental Setup

• Models span closed-source, open-weight understanding-focused, and open-weight unified groups, with the latter evaluated as a proxy for self-generated visual aids.Table 1 reports Direct, Text-CoT, and Visual-CoT results across these model groups.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Main Results

• No model exceeds 20% accuracy with direct image-and-question input; GPT-5 averages 16.5%, and Puzzles average 9.5% versus 16.1% for other categories.The authors characterize MIRA as testing fine-grained visual comprehension and reasoning that current MLLMs handle poorly.Source: §4.2 · p. 7
• Text-CoT offers little to no advantage; Gemini 2.5 Pro and o3 decline by 18.3% and 14.0% relative, respectively.On Puzzles and Causal, Text-CoT harms proprietary-model accuracy by relative averages of 4.2% and 2.6%, respectively.Source: §4.2 · pp. 7–8
• 33.7% average relative score boost across models with annotated Visual-CoT, with gains reported across nearly all models.GPT-5-mini rises from 13.7% to 23.2%; open-weight models average 9.9% with direct input and 13.0% with Visual-CoT.Source: §4.2 · p. 8

4.3 Attempts to Probe the Model Upper-Bound

• Pass@k gains average 15.3% from k=1 to 4 but only 3.0% from k=4 to 8, indicating limited returns from additional sampled reasoning paths.For GPT-5, pass@1-to-pass@8 improves 20.4%, versus 23.6% for GPT-4o.Source: §4.3 · pp. 8–9
• Specialized Text-CoT prompts gain 1.4% on closed-source models and 1.5% on open-weight models, modest compared with Visual-CoT’s 4.7% average gain on closed models.Specialized-prompt gains range from -0.2% to +3.2%.Source: §4.3 · p. 9, Table 2 · p. 9

4.4 Case Study

• In the geometry case, GPT-5 uses a hand-drawn Visual-CoT image to locate the hull intersection and correctly count 4 red dots after text-only reasoning fails.The task requires identifying two convex hulls, locating their overlap, and counting red dots within it.Source: §4.4 · p. 10

• Visual-CoT yields significant gains across tasks, with an average relative improvement of over 33%.Source: §5 · p. 10
• Text-CoT has an intrinsic medium-level limitation on visually intensive tasks, and prompt engineering brings even state-of-the-art models only limited gains.Source: §5 · p. 10
• A significant gap remains between closed-source and open-weight models in their ability to use visual clues effectively.Source: §5 · p. 10

• The appendix includes experimental model settings and prompt settings.Source: Appendix · p. 15
• It includes detailed experimental tables and a dataset showcase.Source: Appendix · p. 15
• It includes a disclosure of large language model use.Source: Appendix · p. 15

• API-based models use endpoint-default decoding settings and a maximum output length of 16,384 tokens.Their versions and checkpoints are organized in Table 3.Source: A Experimental Model Settings · p. 15, Table 3 · p. 15
• Qwen-VL-Max (325B), GLM-4.5V (106B), and Qwen2.5-VL (32B/72B) use a maximum output length of 8,192 tokens and top-p 1.0.Source: A Experimental Model Settings · p. 15
• Bagel in thinking mode and Janus-Pro use the official inference configurations from their respective code repositories.Source: A Experimental Model Settings · p. 15

Text-CoT templates

• Specialized templates provide task-specific reasoning guidance, complementing a general CoT template used as a baseline for text-based reasoning.The specialized templates were designed for each of MIRA’s 20 tasks.Source: B Prompt Settings · p. 16

Task-specific prompts

• The convex-hull prompt asks models to identify extreme points, handle collinearity, order the vertices, and justify the resulting hull.It includes supporting checks such as orientation tests and cross products.Source: (1) Euclidean Geometry (EG) · p. 16
• The physics prompts guide models to determine force directions and magnitudes, resolve vector components, and sum them to obtain net force.They also call for symmetry checks and consideration of special cases.Source: (2) Physics-Based Reasoning (PBR) · p. 18
• Puzzle and transformation prompts cover tasks including cube unfolding, bomb defusal, airplane folding, gear rotation, and dice rolling.The prompts supply task-dependent reasoning cues, such as tracking dice faces after each roll.Source: (3) Abstract Spatial & Logical Puzzles… · p. 19, (4) Causal Transformations (CT) · pp. 20–21

Evaluation levels

• Simulated Visual-CoT supplies annotated intermediate images alongside the question image and text prompt, while Direct Evaluation asks for an answer and Text-CoT prompts step-by-step textual reasoning.The three settings separate direct input, text-based reasoning, and visual clues in the reasoning process.Source: B Prompt Settings · p. 16, (4) Causal Transformations (CT) · p. 21

Evaluation

• The evaluator judges correctness from the final answer against the ground truth and ignores the reasoning and other response text.The prompt directs attention to the answer inside the answer tags and the final-answer portion of the response.Source: (4) Causal Transformations (CT) · pp. 21–22

Detailed results

• The detailed tables supplement the main results presented in Table 1 with sub-category performance breakdowns.The section identifies Tables 4–10 as the detailed results tables.Source: C Detailed Experimental Tables · p. 22
• Detailed results for Convex Hull, Mirror Pattern, and Mirror Clock are provided in Table 4.The caption groups the first two under Euclidean Geometry and Mirror Clock under Physics-Based Reasoning.Source: Table 4 · p. 22
• Table 4 includes Euclidean Geometry results for Convex Hull and Mirror Pattern alongside Physics-Based Reasoning results for Mirror Clock.Source: Table 4 · p. 22

Dataset overview

• 546 multimodal problems span Euclidean Geometry, Physics-Based Reasoning, Abstract Spatial & Logical Puzzles, and Causal Transformations.The tasks are described as requiring intermediate visual reasoning, analogous to drawing to think.Source: D Dataset Showcase · p. 22

Detailed results

• Detailed-results tables cover tasks including Overlap, Unfolded Cube, Billiards, Localizer, Paper Airplane, and Defuse A Bomb.Other listed tables cover Multi-piece Puzzle, Electric Charge, Rolling Dice, Gear Rotation, Trailer Cubes Count, and Trailer Cubes Missing.Source: Table 5 · p. 23, Table 6 · p. 23, Table 7 · p. 24

Representative examples

• To supplement the overview provided in Figure 1 and offer a more intuitive understanding of the dataset, Figures 6–15 showcase several representative examples for each category.Source: D Dataset Showcase · p. 23

Disclosure

• ChatGPT was used to refine language, including grammar, phrasing, and style.The disclosure states that the authors conceived and executed the research concepts, methods, analyses, and conclusions, and are responsible for the manuscript.Source: E Disclosure of Large Language Model Use · pp. 24–25

Supplementary results

• Tables 9 and 10 provide detailed results for Euclidean Geometry and Abstract Puzzles tasks, covering Cubes Count, Cubes Missing, Puzzle, Trailer Cubes Count, and Trailer Cubes Missing.Source: Table 9 · p. 25, Table 10 · p. 25

Illustrative cases

• Illustrative cases include Convex Hull and Cubes Count, Mirror Clock and Cubes Missing, and Mirror Pattern and Overlap.Source: Figure 6 · p. 26, Figure 7 · p. 26, Figure 8 · p. 26