Source-linked AI summary
MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, Hongsheng Li
TL;DR
CoT has advanced LLM reasoning, but its impact on LMMs has not been systematically assessed. MME-CoT provides a six-domain benchmark and fine-grained evaluation suite, finding that reflection improves CoT quality while CoT can harm perception performance and reflection remains inefficient.
Problem
The extent to which CoT benefits multimodal challenges remains an open question, while prior LMM evaluations are insufficiently systematic and thorough.
Method
MME-CoT evaluates LMM CoT reasoning across six domains with metrics for reasoning quality, robustness, and efficiency, using curated annotations and model-plus-human categorization.
Results
Reflection improves CoT quality, Kimi k1.5 achieves the best quality, CoT often degrades perception-task performance, and 30% to 40% of reflection steps fail to help answer questions.
Takeaways & Limitations
CoT should not be treated as a universally beneficial default for LMMs because perception interference and ineffective reflection limit its applicability and efficiency.
Takeaways & Limitations
Kimi k1.5 evaluation uses a 150-question sample because of access limitations, comprising 115 reasoning and 35 perception questions.
Abstract
from arXiv · showhide
Answering questions with Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs), yet its impact on Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth investigation. In this paper, we introduce MME-CoT, a specialized benchmark evaluating the CoT reasoning performance of LMMs, spanning six domains: math, science, OCR, logic, space-time, and general scenes. As the first comprehensive study in this area, we propose a thorough evaluation suite incorporating three novel metrics that assess the reasoning quality, robustness, and efficiency at a fine-grained level. Leveraging curated high-quality data and a unique evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs, uncovering several key insights: 1) Models with reflection mechanism demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and demonstrating the highest quality results; 2) CoT prompting often degrades LMM performance on perception-heavy tasks, suggesting a potentially harmful overthinking behavior; and 3) Although the CoT quality is high, LMMs with reflection exhibit significant inefficiency in both normal response and self-correction phases. We hope MME-CoT serves as a foundation for advancing multimodal reasoning in LMMs. Project Page: https://mmecot.github.io/
1. Introduction
MME-CoT addresses the limited systematic understanding of Chain-of-Thought reasoning in Large Multimodal Models by evaluating reasoning quality, robustness, and efficiency across six multimodal domains. Its analysis finds that reflection improves CoT quality, while CoT can harm perception performance and remain inefficient.
- Benchmark and motivation: MME-CoT evaluates LMM CoT reasoning across math, science, OCR, logic, space-time, and general scenes.The benchmark uses a fine-grained evaluation framework covering reasoning quality, robustness, and efficiency.
- Evaluation suite: The benchmark measures CoT quality through Recall of ground-truth steps and Precision of accurate generated steps.These metrics distinguish informative reasoning from faithful reasoning and address correct answers reached through flawed logic or guessing.
- Evaluation suite: CoT robustness is evaluated by comparing direct-answer and step-by-step prompting on perception and reasoning tasks using Stability and Efficacy.Stability measures potential degradation on perception tasks, while Efficacy measures improvement on complex reasoning tasks.
- Evaluation suite: CoT efficiency is assessed with Relevance Rate and Reflection Quality, which measure useful content and whether reflection advances correctness.The efficiency analysis targets the accuracy–computational-cost trade-off associated with long reasoning and reflection processes.
- Key findings: Reflection improves CoT quality, with QVQ achieving an F1 Score of 62.0%, surpassing Qwen2-VL-72B by 6.8%.Kimi k1.5 beats GPT-4o and achieves the best quality results.
- Key findings: CoT often interferes with perception tasks, with InternVL2.5-8B showing a 6.8% degradation when CoT is applied.Around 30% to 40% of reflection steps fail to help answer questions, especially in general scenes, space-time, and OCR tasks.
2. Dataset Curation
MME-CoT organizes multimodal questions into reasoning and perception tasks across six domains and 17 subcategories, using model-based and human assessment for categorization and detailed annotations for CoT evaluation.
- Data Composition and Categorization: MME-CoT spans six major domains and 17 subcategories, covering both reasoning and perception questions.Reasoning tasks require multi-step logical inference, whereas perception tasks primarily test visual recognition or minimal reasoning.
- Data Composition and Categorization: The benchmark distinguishes reasoning-dominant from perception-focused subcategories by comparing model performance with and without CoT prompting.GPT-4o and Qwen2-VL-7B provide preliminary classifications, which are then combined with human assessment.
- Data Annotation and Review: Reasoning questions receive key-step annotations and reference image captions to support fine-grained CoT evaluation.Key steps include inference conclusions and image captions, with the final answer included as a concluding inference.
- Data Annotation and Review: Annotators verify GPT-4o-generated rationales and captions, independently solving cases where generated rationales are unreasonable and recording all methods for multiple-solution problems.Annotations retain only core conclusions and relevant visual-element descriptions.
3. CoT Evaluation Strategy
MME-CoT evaluates CoT reasoning in LMMs beyond final-answer accuracy by separately measuring quality, robustness, and efficiency. Its strategy uses annotated solution steps, step partitioning, prompt comparisons, and analysis of relevance and reflection validity.
- Evaluation motivation: Existing benchmarks mainly judge final answers, leaving CoT processes insufficiently evaluated for informativeness and reasoning validity.MME-CoT addresses this gap with a holistic evaluation suite covering correctness, stability, efficacy, relevance, and reflection quality.
- CoT quality evaluation: Recall measures how many annotated solution steps appear in a response, selecting the most-matched method when questions have multiple solution paths.The metric is intended to assess informative progress and reduce the chance that a correct answer results from random guessing.
- CoT quality evaluation: Precision evaluates the faithfulness of image-caption and logical-inference steps after GPT-4o partitions predictions into those categories and background information.The two step types are emphasized because visual reasoning is characterized primarily by interleaved image captions and logical inferences; F1 combines precision and recall for CoT quality.
- CoT robustness evaluation: Robustness compares direct-answer and step-by-step prompts on perception and reasoning tasks using stability for perception and efficacy for reasoning.The strategy tests whether CoT interferes with perception while measuring its performance difference on reasoning tasks.
- CoT efficiency evaluation: Efficiency evaluates long reasoning processes through relevance rate and reflection quality, identifying irrelevant content and judging whether reflection steps correct mistakes or add new insight.Relevant steps must work toward solving the question, while valid reflections either identify previous errors or verify conclusions with new insight.
4. Experiments
The experiments evaluate LMM Chain-of-Thought across quality, robustness, and efficiency, revealing gains from reflection alongside perception degradation and inefficient reasoning.
- Evaluation Setup: MME-CoT evaluates CoT using precision, recall, relevance, robustness, and reflection-oriented efficiency measures.The evaluation reports precision, recall, and relevance for logical inference and image-caption steps, plus direct-versus-CoT robustness results.
- CoT Quality: Kimi k1.5 achieves the highest CoT quality, while larger open-source models generally perform better.QVQ surpasses Qwen2-VL-72B by 5.8% after reasoning-focused fine-tuning, and Qwen2-VL-72B outperforms other open-source models without reflection.
- Robustness: Most models suffer degraded perception-task performance under CoT, with InternVL2.5-8B declining by 6.8%.Only LLaVA-OV-72B and InternVL2.5-8B-MPO achieve a modest positive robustness score of 0.3% among models providing direct answers.
- Efficiency: Long CoT can include irrelevant or repeated content and may omit key solution steps despite high precision.Reflection quality is also limited: QVQ and Virgo score about 60%, while more than 25% of Kimi k1.5 reflection steps are invalid.
- Reflection Errors: The reflection analysis identifies ineffective reflection, incompleteness, repetition, and interference as four error types.These errors respectively involve persistent incorrect adjustments, unfinished approaches, restated content, and introducing errors after an initially correct conclusion.
5. Conclusion
The conclusion presents MME-CoT as a comprehensive benchmark and evaluation suite for multimodal CoT reasoning. It reports critical flaws in tested open-source models and positions the benchmark as a tool for measuring progress.
- Conclusion: MME-CoT evaluates Chain-of-Thought reasoning in Large Multimodal Models across six categories of visual reasoning tasks.The benchmark is designed to cover most scenarios of visual reasoning tasks.
- Conclusion: The paper introduces a three-metric CoT evaluation suite to examine multimodal reasoning processes.The conclusion describes the suite as a way to gain a thorough understanding of the reasoning process.
- Conclusion: Systematic evaluation identifies critical flaws in all tested open-source models.The conclusion frames these findings as insights into issues within current state-of-the-art Large Multimodal Models.
- Conclusion: MME-CoT is presented as a tool for measuring progress and identifying improvement areas in multimodal AI development.The conclusion states that the benchmark can support development of more sophisticated multimodal AI systems.
A.2. Reasoning Evaluation
Existing reasoning evaluations either focus on text-only scenarios or concentrate multimodal benchmarks on specific domains, leaving broader visual CoT assessment limited.
- Existing Evaluation: Text-only reasoning evaluators assess dimensions such as correctness and informativeness but do not address visual reasoning challenges.ROSCOE and ReCEval are cited as examples of text-only evaluation approaches.
- Research Gap: These limitations motivate broader evaluation of Chain-of-Thought reasoning in multimodal settings.The cited literature distinguishes the scope of text-only evaluators from that of existing multimodal benchmarks.
- Existing Evaluation: Multimodal reasoning benchmarks have predominantly focused on mathematics and selected domain-specific tasks.MathVista and MathVerse are cited as mathematical and logical reasoning benchmarks.
B.1. Data Source Distribution
MME-CoT draws its data from 15 source sets, providing a distributed basis for its multimodal reasoning benchmark.
- Data Sources: MME-CoT’s data-source distribution comprises 15 benchmark sets.The listed sources include MathVerse, MMMUPro, OlympiadBench, MMT-Bench, and MuirBench.
- Data Sources: The source collection spans mathematical, multimodal, scientific, search, and visual reasoning datasets.Examples include MathVerse, SciVerse, MMSearch, CharXiv, and MMT-Bench.
- Data Sources: Figure 9 visualizes the distribution of MME-CoT’s data sources.The figure is specifically captioned as the data source distribution of MME-CoT.
B.2. Preliminary Categorization Result
The preliminary categorization results report accuracy breakdowns for MMT-Bench, MUIRBench, and OlympiadBench across their respective subcategories.
- MMT-Bench accuracy is organized by diverse visual-language subcategories, including action understanding, counting, OCR, temporal reasoning, and visual comprehension.
- MUIRBench accuracy is similarly categorized across tasks such as action understanding, counting, scene understanding, visual grounding, and visual retrieval.
- OlympiadBench accuracy is reported for mathematics and physics subcategories.
C. Error Analysis
The error analysis presents reflection failures in visual reasoning, including inconsistent geometric reasoning, alternative but unsupported interpretations, and answers that remain wrong after reflection.
- In image-ordering tasks, reflection may switch between horizontal and vertical scene interpretations, yielding different sequences.
- One reflection concludes that none of the answer choices matches the inferred image sequence and selects that option.
- A separate example explicitly records a wrong answer after reflection.
- In a time-based weight-gain problem, the reasoning first considers daily conversion but then recognizes that month-based measurement is sufficient.
- Reflection can produce conflicting geometric conclusions, alternating between 105° and 140° for the same angle.
D. More Qualitative Examples
The qualitative examples illustrate correct and incorrect multimodal reasoning across geometry, healthiness judgments, and puppy counting, including cases where visual interpretations determine the answer.
- Similar-triangle reasoning uses EF = 3, BC = 15, and AB = 45 to obtain DE = 9.
- Another healthiness example likewise selects Object 1, described as fruit and nut salad, over an apparently fried or roasted chicken dish.
- One healthiness example identifies Object 1 as healthier than Object 2 based on the apparent fruit-and-nut contents versus fried food.
- The puppy-counting example finds zero puppies sticking out their tongues and selects answer A.
E. Detailed Evaluation Setup
The evaluation setup reformats multimodal solutions into atomic steps and judges correctness, relevance, and reflection quality using ground-truth annotations and explicit criteria.
- Solutions are divided into logical inference, image observation or description, and background information steps while preserving order and content.
- Correctness judgments require image observations to match ground truth and logical conclusions to match or be directly entailed by it.
- Relevance is evaluated independently of correctness, counting steps that meaningfully attempt to solve the question even when their logic is wrong.
- Reflections are identified through reconsideration cues and judged correct only when they reach accurate conclusions using new insights to correct or verify prior reasoning.