Source-linked AI summary
M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, Wanxiang Che
TL;DR
Existing MCoT benchmarks often lack genuine visual reasoning, multi-step reasoning, and important domains, making their apparent progress incomplete. The paper introduces M3CoT to address these gaps and evaluates representative approaches; the results show that VLLMs still struggle and remain substantially below human performance. M3CoT is presented as a challenging foundation for further multi-domain, multi-step, multi-modal CoT research.
Problem
Existing MCoT benchmarks lack visual-modal reasoning, multi-step visual reasoning, and domains such as commonsense and mathematics.
Method
M3CoT removes samples answerable without images, selects multi-step multimodal samples through expert annotation, and augments commonsense and mathematics domains.
Results
VLLMs still struggle on M3CoT and remain substantially behind human performance despite strong results on earlier MCoT benchmarks.
Takeaways & Limitations
M3CoT provides a challenging benchmark for reassessing multimodal CoT progress and motivating research on multi-domain, multi-step, multi-modal reasoning.
Takeaways & Limitations
Manual annotation may introduce bias, and the dataset is English-only rather than multilingual.
Abstract
from arXiv · showhide
Multi-modal Chain-of-Thought (MCoT) requires models to leverage knowledge from both textual and visual modalities for step-by-step reasoning, which gains increasing attention. Nevertheless, the current MCoT benchmark still faces some challenges: (1) absence of visual modal reasoning, (2) single-step visual modal reasoning, and (3) Domain missing, thereby hindering the development of MCoT. Motivated by this, we introduce a novel benchmark (M$^3$CoT) to address the above challenges, advancing the multi-domain, multi-step, and multi-modal CoT. Additionally, we conduct a thorough evaluation involving abundant MCoT approaches on Vision Large Language Models (VLLMs). In addition, we highlight that the current VLLMs still struggle to correctly reason in M$^3$CoT and there remains a large gap between existing VLLMs and human performance in M$^3$CoT, despite their superior results on previous MCoT benchmarks. To our knowledge, we take the first meaningful step toward the multi-domain, multi-step, and multi-modal scenario in MCoT. We hope that M$^3$CoT can serve as a valuable resource, providing a pioneering foundation in multi-domain, multi-step, multi-modal chain-of-thought research.
1 Introduction
The paper argues that existing MCoT benchmarks overestimate progress because they often omit visual reasoning, require only single-step visual reasoning, and lack important domains. It introduces M3CoT as a multi-domain, multi-step, multi-modal benchmark and evaluates representative approaches on it.
- Motivation: Existing MCoT benchmarks can overestimate progress because their samples are often too simple.The paper identifies this simplicity through benchmark analyses of reasoning complexity and diversity.
- Motivation: Current benchmarks lack genuine visual reasoning, often allowing answers from text alone or from a single visual step.The paper contrasts absent visual reasoning with cases requiring only one visual object before introducing multi-step visual reasoning.
- Motivation: Existing benchmarks also omit commonsense and mathematics, limiting domain coverage for comprehensive MCoT evaluation.The paper identifies domain missing as a third major drawback alongside insufficient visual reasoning complexity.
- Benchmark: The benchmark removes samples answerable without images, manually selects multi-step multimodal samples, and augments commonsense and mathematics data.Its construction combines filtering, expert annotation, and LLM-guided domain augmentation.
- Evaluation: Evaluation finds that VLLMs still struggle on M3CoT and remain behind human performance.The paper reports this as a central takeaway from evaluating representative MCoT approaches.
- Benchmark: M3CoT is introduced as the first benchmark targeting multi-domain, multi-step, multi-modal CoT.The benchmark is intended to address the three weaknesses identified in existing datasets.
2 Problem Formalization
The formalization defines multi-step multi-modal CoT over an image, question, context, and answer options. The model constructs a textual prompt, generates a step-wise rationale whose selected steps use visual input, and then predicts the final answer.
- Task Definition: The task input consists of an image I, question Q, context C, and options O = {o_1, ..., o_n}.These components define the multimodal reasoning scenario.
- Prompt Construction: A prompt T is constructed from the textual question, context, and options using a prompting method.Prompt(·) denotes the method that converts textual inputs into an instruction format.
- Rationale Generation: The model generates a rationale R_m = {S_1, ..., S_m} one step at a time, conditioned on the preceding rationale.Each step is generated using the prompt and previously generated steps.
- Rationale Generation: Steps requiring multimodal reasoning condition on image I, while other steps condition only on T and the preceding rationale.The set S identifies the image-dependent reasoning steps.
- Task Definition: Reasoning is classified as multi-step and multi-modal when at least two rationale steps require multimodal reasoning.The formal criterion is |S| ≥2.
- Answer Prediction: The final answer Y is selected from the options after rationale generation.The formalization defines answer prediction from the generated rationale.
3 Dataset Annotation
M3CoT annotation combines automated filtering, expert verification, domain augmentation, and quality assurance. The workflow targets visual dependence, multi-step multimodal reasoning, missing domains, and reliable annotations.
- Workflow: The annotation workflow covers visual-reasoning sample removal, multi-step sample construction, domain augmentation, and quality assurance.These stages are organized as sections 3.1 through 3.4.
- Absence of Visual Modal Reasoning Sample Removal: Samples without images are automatically removed, then experts verify whether image content is genuinely required for MCoT.Manual review addresses image-containing samples that remain answerable without visual reasoning.
- Multi-step MCoT Sample Construction: Samples with rationales shorter than two steps are automatically filtered before expert annotation.This reduces annotation burden and targets multi-step reasoning.
- Multi-step MCoT Sample Construction: Experts first inspect textual context and rationales without images, then use the images to confirm multi-step reasoning across modalities.The staged review tests whether visual information is needed repeatedly during reasoning.
- MCoT Domain Augmentation: M3CoT augments missing mathematics and commonsense data using MATH and Sherlock, respectively.Mathematics samples receive generated options and rendered images, while commonsense samples receive generated questions, options, and answers from visual clues.
- Quality Assurance: Quality assurance uses annotator testing, rationale review, two rechecks, agreement by at least two annotators, and reports a kappa coefficient of 0.85.Only annotators achieving at least 80% accuracy on a 100-sample onboarding test proceed.
4 Data Analysis
M3CoT contains 7,863 training, 1,108 validation, and 2,358 test samples, with longer and more complex rationales than ScienceQA. Its analyses also report diverse image semantics and a broader rationale-step distribution.
- Basic Statistics: M3CoT is split into 7,863 training, 1,108 validation, and 2,358 test samples.The dataset is randomly partitioned into three subsets.
- Basic Statistics: 294 average rationale length in M3CoT exceeds ScienceQA’s 48.The reported lengths indicate greater reasoning intricacy in M3CoT.
- Multi-modal Diversity: M3CoT spans diverse image types and a broader semantic space than ScienceQA.Figure 4 analyzes image diversity and represents semantic-space coverage with point area.
- Rationale Diversity: 10.9 average rationale steps in M3CoT exceeds ScienceQA’s 2.5, OKVQA’s 3.0, MMMU’s 1.0, and VCR’s 1.0.The M3CoT rationale-step distribution is also described as more uniformly distributed.
5 Experiments
Experiments show that M3CoT exposes substantial weaknesses in current VLLMs: multi-step multimodal reasoning remains difficult, while fine-tuning is more effective than prompting, tool usage, or standard ICL.
- Main Results: Open-source VLLMs lag GPT4V by at least 7.98% on M3CoT, while GPT4V still substantially trails human performance.These results indicate persistent gaps in both open-source and leading proprietary models.
- Main Results: Zero-shot multimodal CoT benefits only VLLMs with at least 13B parameters; smaller models show no reasoning improvement.The findings associate larger model scale with observable emergent capabilities in this setting.
- Multi-step Analysis: 29.06%: M3CoT performance is at least 29.06% lower than ScienceQA performance, and accuracy declines as reasoning steps increase.The analysis attributes the gap to the greater complexity of multi-step reasoning.
- Multi-step Analysis: Higher rationale quality and more averaged multimodal interaction steps correlate positively with M3CoT reasoning performance.Rationale quality is evaluated across five dimensions, while interaction steps are identified through image–reasoning-step similarity.
- Tool Usage: Multimodal tool usage on text-only LLMs performs poorly, with some systems 28.21% below GPT4V and even below random baseline.The paper attributes this to planning that cannot observe visual information during tool selection and use.
- ICL and Fine-tuning: Text-only ICL examples fail to significantly improve multimodal reasoning, while image–text interleaving can degrade performance; fine-tuning is more effective.Finetuned VLMs range from 44.85% accuracy, exceeding zero-shot open-source VLLMs capped at 38.86%, and some surpass Gemini’s 47.50%.
6 Related Work
Related work extends textual Chain-of-Thought to multimodal reasoning through benchmarks, rationale distillation, knowledge integration, and multimodal interaction. M3CoT builds on this literature by targeting multi-step reasoning across broader domains and evaluating diverse approaches.
- Multimodal Chain-of-Thought: Prior work established multimodal CoT through ScienceQA, two-stage reasoning, rationale distillation, knowledge graphs, and multimodal interaction.These approaches extend step-by-step reasoning beyond text-only language models.
- M3CoT: M3CoT differs from previous work by targeting multi-step multimodal reasoning across a broader range of domains.The paper also conducts comprehensive experiments and reports takeaways for future research.
7 Conclusion
The work introduces M3CoT as a benchmark for multi-domain, multi-step, multi-modal chain-of-thought scenarios and analyzes existing MCoT methodologies to expose their limitations.
- M3CoT targets multi-domain, multi-step, and multi-modal chain-of-thought scenarios through a detailed benchmark development process.
- The authors evaluate abundant multi-modal CoT methodologies on M3CoT to characterize limitations in existing frameworks.
Limitations
The benchmark has limitations from manual annotation subjectivity and English-only data, while future work will prioritize open-source models because some models may be discontinued or retired.
- Manual annotation may introduce biases because of unavoidable human subjectivity, potentially affecting data reliability.
- The dataset is limited to English because regional and cost restrictions prevented inclusion of multilingual backgrounds.
- Future work will focus more on open-source models because some models may be discontinued or retired.
Ethical Considerations
The study uses open-source datasets and recruits compensated university participants who provide informed consent for annotation and rechecking work.
- The data come from open-source datasets that are freely available for academic research and intended to support ethical data use.
- Participants are recruited from universities, must meet English-proficiency requirements, and may introduce national biases despite efforts to minimize differences.
- Annotators provide informed consent and receive compensation above the local minimum wage for their work.
- The process includes onboarding, hourly annotation, rechecking, and participation by six experts and three students.
A.1 Statistical Analysis of Existing Datasets
The analysis examines deficiencies in existing multi-modal reasoning datasets and describes procedures for sampling, filtering, annotating, augmenting, and evaluating data for M3CoT.
- Statistical analysis: At least 79% of data across the examined benchmarks lack sufficient multi-step reasoning capability.
- Statistical analysis: A stratified random sample comprising 20% of each dataset is manually inspected to estimate multi-step multi-modal CoT proportions.
- Statistical analysis: Manual review categorizes sampled points for multi-step reasoning, supporting findings that existing datasets need more complex reasoning tasks.
- Data processing: Samples with fewer than two rationale steps are automatically filtered, while multi-step and single-step cases still receive manual evaluation.
- Domain augmentation: MATH is augmented with generated multiple-choice options and visual representations so its textual mathematical content can support multimodal evaluation.
- Domain augmentation: The study uses structured prompting to make models consider multiple visual clues simultaneously, while rationale rewriting addresses poor expression in ScienceQA explanations.
B.3.2 Tool Usage Error Analysis
Tool-use errors in M3CoT arise from mistakes in interpreting image information and logic, causing poor tool selection, redundant tool chains, and task failure in complex workflows.
- Image-information and logical mistakes are identified as initial challenges in the tool-usage analysis.
- Lack of visual interaction during tool planning leads to frequent tool-selection errors in multi-tool, multi-step scenarios.The analysis illustrates this issue with a GPT4V response on the commonsense domain and a HuggingGPT response on commonsense.
- Incorrect tool planning or selection can cascade through the process and culminate in complete failure of the intended task.The analysis cites a HuggingGPT example for tool-selection failure.
- Tool-chain redundancy is illustrated through an IdealGPT response on the commonsense domain.
- The analysis includes GPT4V examples from mathematics and science, including a geometry question requiring altitude identification and a building-function question.