MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency

• MME-CoT addresses insufficiently systematic prior evaluation with a human-verified benchmark spanning math, science, OCR, logic, space-time, and general scenes.Its evaluation suite examines CoT reasoning quality, robustness, and efficiency.Source: §1 · pp. 2–3
• CoT quality is measured with Recall for ground-truth reasoning-step coverage and Precision for the accuracy of generated steps.These metrics address the limitation of outcome-only evaluation, which can mistake correct answers reached through flawed logic or random guessing for sound reasoning.Source: §1 · p. 3
• Robustness evaluation compares direct-answer and step-by-step prompting on perception and reasoning tasks, measuring CoT Stability and Efficacy.Stability checks for harm to direct perception performance, while Efficacy measures gains on complex reasoning tasks.Source: §1 · p. 3
• Efficiency is assessed through Relevance Rate, the share of generated content that helps answer the question, and Reflection Quality, whether reflection steps move toward correctness.This evaluation addresses the accuracy-versus-computational-cost trade-off raised by long CoT and reflection processes.Source: §1 · p. 3
• QVQ achieves F1 Score of 62.0%, surpassing Qwen2-VL-72B by 6.8%; Kimi k1.5 beats GPT-4o and achieves the best quality.The paper reports that fine-grained reflection greatly enhances CoT quality.Source: §1 · p. 3
• Most models are interfered with by CoT on perception tasks, with InternVL2.5-8B showing the worst-case 6.8% degradation.The authors interpret this as harmful overthinking that can impede using CoT by default.Source: §1 · p. 3
• Around 30% to 40% of reflection steps fail to help answer questions, especially in general scenes, space-time, and OCR tasks.The paper also notes that models can be distracted by image content and that not all long-CoT steps contribute to answering.Source: §1 · p. 3

2.1. Data Composition and Categorization.

• MME-CoT covers six domains and 17 subcategories, excludes questions requiring specialized knowledge, and separates multi-step reasoning tasks from visual-recognition or minimal-reasoning perception tasks.The benchmark includes commonsense scenarios alongside problems requiring rigorous logic; it distinguishes task types that existing benchmarks often conflate.Source: §2.1 · p. 3
• A two-stage classification process uses GPT-4o and Qwen2-VL-7B CoT-versus-direct performance to guide categorization, then expert annotators finalize question labels; the dataset contains 1,130 questions and 3,865 key-step annotations.CoT superiority indicates reasoning-dominant subcategories, while comparable or lower CoT performance suggests perception-focused content or limited model reasoning capability.Source: §2.1 · p. 4

2.2. Data Annotation and Review

• For reasoning questions, GPT-4o generates rationales and image captions, and annotators use these to provide key intermediate steps, independently solving cases where its rationale is unreasonable.Providing the ground-truth answer with the question yields more accurate rationales than question-only prompting; steps comprise inference conclusions and image captions, including the final answer, reduced to core conclusions and relevant visual descriptions.Source: §2.2 · p. 4
• Annotators provide all possible methods for problems with multiple solutions and verify and correct reference-caption details.Source: §2.2 · p. 4

3.1. CoT Quality Evaluation

• CoT quality is the F1 score of recall over annotated key steps and precision over image-caption and logical-inference steps, measuring informativeness and faithfulness.For questions with multiple methods, recall is computed on the method with the most matched key steps; GPT-4o partitions predictions and evaluates step correctness.Source: §3.1 · pp. 4–5, Figure 4 · p. 5, Figure 5 · p. 6

3.2. CoT Robustness Evaluation

• Robustness compares direct-answer and CoT prompt accuracy, defining stability on perception tasks, where CoT should not hurt, and efficacy on reasoning tasks, where CoT should help.GPT-4o mini extracts final answers for comparison with ground truth; negative stability indicates CoT may interfere with correct perception judgments.Source: §3.2 · pp. 6–7

3.3. CoT Efficiency Evaluation

• Efficiency measures the relevance of reasoning steps and the validity of reflective steps; the overall relevance rate uses α = 0.8.A reflection is valid if it correctly identifies an earlier mistake or verifies a prior conclusion with new insight; redundant or misleading reflections are not valid.Source: §3.3 · pp. 7–8, Figure 7 · p. 7

4.1. Experiment Setup

• The evaluation covered non-reasoning, reasoning-targeted, and reflection-capable LMMs, with Kimi k1.5 evaluated on a 150-question sample.The sample contained 115 reasoning and 35 perception questions; the study compared direct and step-by-step prompts and used GPT-4o mini for direct evaluation and GPT-4o for other criteria.Source: §4.1 · p. 8

4.2. Quantitative Results

• Kimi k1.5 achieved the highest CoT quality, GPT-4o the highest robustness score, and InternVL2.5-8B the highest relevance at 98.4%.GPT-4o led across recall metrics, while Kimi k1.5 led in precision evaluations.Source: §4.2 · p. 9
• Reflection-capable models largely benefited CoT quality: QVQ surpassed Qwen2-VL-72B by 5.8% in F1 and 2.9% in precision, while Kimi k1.5 surpassed GPT-4o in quality.Larger open-source models generally performed better; Qwen2-VL-72B also outperformed other open-source models without reflection, including InternVL2.5-78B-MPO.Source: §4.2 · p. 9
• Long CoT could skip intermediate steps despite correct answers and lose focus in image descriptions, with irrelevant exhaustive captions common in general scenes, space-time, and OCR.Virgo and QVQ outperformed GPT-4o in direct evaluation but lagged in recall; long-CoT models also tended to have lower relevance, potentially slowing inference.Source: §4.2 · pp. 9–10, Table 3 · p. 9
• CoT impaired perception-task performance in most models, with InternVL2.5-8B dropping 6.8%; only LLaVA-OV-72B and InternVL2.5-8B-MPO had modest positive scores of 0.3%.The authors describe the negative stability scores as evidence of inconsistency and potential overthinking, cautioning against CoT as a default answering strategy.Source: §4.2 · pp. 9–10
• Larger parameter counts tended to improve reasoning efficacy: Qwen2-VL-7B fell 4.8% with CoT, whereas Qwen2-VL-72B improved 2.4%.The same general pattern appeared across LLaVA-OV and InternVL2.5-MPO model sizes.Source: §4.2 · p. 10

4.3. Error Analysis

• Reflection often fails to help: QVQ and Virgo achieve only about 60% reflection-quality scores (about 40% of attempts fail to aid accuracy), while over 25% of Kimi k1.5 reflection steps are invalid.Ineffective reflection—continuing incorrect adjustments after an incorrect conclusion—is the most common error type.Source: §4.2 · p. 10, §4.3 · p. 10

5. Conclusion

• MME-CoT covers six categories of visual reasoning and uses three metrics to evaluate LMM Chain-of-Thought reasoning.The paper presents the benchmark as a tool for understanding reasoning processes and assessing current models.Source: §5 · p. 10
• The evaluation identifies critical flaws in all tested open-source models, while MME-CoT offers a tool for measuring progress and identifying areas for improvement.The conclusion does not specify the flaws in this passage.Source: §5 · pp. 10–11

Appendix Overview

• The appendix overview lists related work, additional dataset details, error analysis, qualitative examples, and evaluation prompts.Source: Appendix Overview · p. 14

A.1. Large Multimodal Models

• Large Multimodal Models build on language and vision models to process multiple kinds of visual input.The passage situates LMM development within the growth of multimodal AI and cites Radford et al. (2021), Li et al. (2022), OpenAI (2023; 2024b), Liu et al. (2023), Zhu et al. (2023), Lin et al. (2023), and Qwen Team (2024).Source: §A.1 · p. 14
• Closed-source GPT-4o (OpenAI, 2024b) shows strong visual understanding and reasoning, while open-source projects expanded multimodal instruction tuning and application scenarios.Early LLaVA (Liu et al., 2023), LLaMA-Adapter (Zhang et al., 2024b), and MiniGPT-4 (Zhu et al., 2023) combined frozen CLIP image encoders with LLMs; later InternVL2 (Chen et al., 2024c), Qwen2-VL, SPHINX (Gao et al., 2024; Lin et al., 2023), and MiniCPM-V (Yao et al., 2024b) broadened visual instruction data and applications.Source: §A.1 · p. 14
• Recent LMM reasoning work builds on o1 (OpenAI, 2024a), exploring mixed preference optimization, collective model knowledge, and iterative self-reflection and verification for multi-step CoT.The passage attributes mixed preference optimization to Wang et al. (2024d), collective knowledge for identifying reasoning paths to Yao et al. (2024a), and o1-like behaviors to Team (2024) and Du et al. (2025).Source: §A.1 · p. 14

A.2. Reasoning Evaluation

• Text-only methods such as ROSCOE (Golovneva et al., 2022) and ReCEval (Prasad et al., 2023) assess reasoning dimensions but omit visual challenges and long-CoT efficiency and reflection quality.The section identifies these as additional considerations not adequately covered by existing evaluation methods.Source: §A.2 · p. 14
• MathVista, MathVerse (Zhang et al., 2024c), and OlympiadBench focus multimodal reasoning evaluation on mathematics and related scientific problems.MathVerse tests interpretation of graphical representations, while OlympiadBench adds Olympiad-level mathematics and physics; general-scene reasoning remains relatively unexplored.Source: §A.2 · p. 14
• Most existing multimodal benchmarks evaluate final answers alone, offering limited insight into the intermediate steps through which models reach conclusions.M³CoT and SciVerse add commonsense tasks alongside scientific reasoning and knowledge-based assessment, but the section notes the broader process-evaluation gap.Source: §A.2 · p. 14

B.1. Data Source Distribution

• MME-CoT combines 15 dataset sets, including MathVerse (Zhang et al., 2024c), MMMUPro (Yue et al., 2024), and OlympiadBench.Other named sources include MMT-Bench (Ying et al., 2024), MuirBench (Wang et al., 2024a), ml-rpm-bench (Zhang et al., 2024e), MMSearch (Jiang et al., 2024), CharXiv (Wang et al., 2024e), and SciVerse.Source: §B.1 · p. 15

B.2. Preliminary Categorization Result

• Table 4 reports MMT-Bench accuracy across subcategories including action understanding, counting, OCR, temporal reasoning, and visual reasoning.The caption defines 22 subcategory abbreviations, spanning action, perception, reasoning, and generation tasks.Source: Table 4 · p. 16
• Table 5 reports MUIRBench accuracy across 12 subcategories, including action understanding, counting, scene understanding, and visual retrieval.Source: Table 5 · p. 16
• Table 6 reports OlympiadBench accuracy for its mathematics and physics subcategories.Source: Table 6 · p. 16

C. Error Analysis

• Figure 10 illustrates examples of the reflection error types identified by the authors.Source: Figure 10 · p. 17, C. Error Analysis · p. 17

• Figures 11–16 present examples of precision and recall evaluation.Source: Figure 11 · p. 18, Figure 12 · p. 19, Figure 13 · p. 20
• Figure 17 also presents a precision and recall evaluation example.Source: Figure 17 · p. 24
• Figures 18–20 present examples of relevance rate evaluation.Source: Figure 18 · p. 25, Figure 19 · p. 26, Figure 20 · p. 27
• Figure 21 presents an example of reflection quality evaluation.Source: Figure 21 · p. 28

E.1. CoT Quality Evaluation Prompts

• For CoT quality, the prompt restructures solutions into ordered steps—logical inference, direct image observation, or background information—without adding, omitting, or reordering content.Steps should be atomic and nonduplicative; initial analysis is background information, while determining the final answer is a logical inference.Source: §E.1 · p. 30
• Each step is judged against ground truth or for reasonableness, with labels distinguishing Match, Reasonable, Wrong, and background-step N/A.Image observations are checked for matching key elements, while inferences must match or be entailed by ground truth; otherwise, valid noncontradictory reasoning may count as reasonable.Source: §E.1 · pp. 30–31

E.2. CoT Efficiency Prompt

• Efficiency scoring marks a step relevant when at least 75% of its content concerns solving the question, regardless of whether its reasoning is correct.The prompt preserves the solution’s original order and content while dividing it into distinct steps.Source: §E.2 · pp. 31–32
• Planning, verification, useful image depictions, option analysis, and even repeated solution-directed steps count as relevant; tangential imagery or thought is irrelevant.Relevance depends on whether the step helps obtain the solution, not on correctness.Source: §E.2 · p. 32
• Reflection evaluation identifies reconsiderations or doubts and judges them correct only when they reach accurate conclusions using new insight to find or verify a mistake.Repetition, a wrong conclusion, and proposing but not executing a new analysis method are invalid reflection types.Source: §E.2 · p. 33

E.3. Direct Evaluation Prompt

• For direct evaluation, the prompt extracts a multiple-choice letter or a non-multiple-choice final value or phrase, returning Z when options conflict or no final answer is given.Multiple-choice output is a single uppercase option character when valid; non-multiple-choice answers may be embedded in the final calculation or inference.Source: §E.3 · p. 34
• A separate consistency prompt compares the extracted model answer with the standard answer, requiring matching option content or equivalent non-multiple-choice meaning and outputting 1 or 0.For multiple-choice questions, an option-letter answer must match the standard answer; equivalent units such as 0.5m and 50cm count as consistent for other questions.Source: §E.3 · p. 35