Source-linked AI summary
Multimodal Chain-of-Thought Reasoning in Language Models
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, Alex Smola
TL;DR
CoT research has largely focused on language, while multimodal reasoning requires integrating vision and text. Multimodal-CoT separates rationale generation from answer inference, achieving state-of-the-art ScienceQA performance with a model under 1 billion parameters and showing benefits for hallucination mitigation and convergence.
Problem
Existing CoT studies largely focus on language, with little consideration of multimodal scenarios.
Method
Multimodal-CoT fine-tunes language models in a two-stage framework that fuses vision and language representations, generates rationales, and then infers answers.
Results
Multimodal-CoT achieves state-of-the-art performance on ScienceQA, with analyses reporting reduced hallucination and enhanced convergence speed.
Takeaways & Limitations
The approach is reported as effective across tasks and backbone models for multimodal CoT reasoning.
Takeaways & Limitations
The work focuses on language and vision modalities and identifies map understanding as requiring additional commonsense signals.
Abstract
from arXiv · showhide
Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.
1 Introduction
Existing CoT research has largely focused on language, motivating Multimodal-CoT for multimodal reasoning with vision and language. The proposed two-stage framework generates multimodal rationales before inferring answers, addressing hallucinated rationales in smaller models.
- Existing CoT studies largely focus on language, with little consideration of multimodal scenarios.
- Multimodal-CoT decomposes multimodal problems into intermediate rationales and then infers the answer.
- Smaller language models tend to generate hallucinated rationales that can mislead answer inference.
- The framework separates rationale generation from answer inference so answers can leverage rationales based on multimodal information.
- The approach is evaluated across multimodal reasoning benchmarks and is designed for language and vision modalities.
- The method achieves state-of-the-art performance on ScienceQA and is reported as effective across tasks and backbone models.
2 Background
Prior CoT research uses prompting or fine-tuning to elicit intermediate reasoning, but smaller models often produce illogical chains that make multimodal reasoning especially challenging.
- CoT Reasoning with LLMs: CoT reasoning is commonly elicited through Zero-Shot-CoT and Few-Shot-CoT prompting paradigms.
- CoT Reasoning with LLMs: Few-Shot-CoT often outperforms Zero-Shot-CoT when supplied with effective demonstrations.
- CoT Reasoning with LLMs: Research on Few-Shot-CoT optimizes demonstrations by improving their quality, selection, diversity, or reasoning complexity.
- CoT Reasoning with LLMs: Other approaches optimize reasoning chains through decomposition into sub-problems solved sequentially or with diverse structures.
- Eliciting CoT Reasoning by Fine-Tuning Models: Fine-tuning studies include explanation-only CoT, knowledge distillation from larger teachers, and context-aware prompt synthesis.
- Eliciting CoT Reasoning by Fine-Tuning Models: Models under 100 billion parameters tend to produce illogical CoT, making effective multimodal reasoning harder than direct answer generation.
3 Challenge of Multimodal-CoT
The section finds that small language models struggle with chain-of-thought reasoning because generated rationales can mislead answer inference. A two-stage Multimodal-CoT framework incorporating vision features improves rationale quality, mitigates hallucination, and supports more accurate inference.
- Challenge setup: 1B-models remain difficult to use for multimodal chain-of-thought reasoning despite their suitability for consumer-grade GPU deployment.The work investigates how to overcome this challenge in models under 1 billion parameters.
- Challenge setup: 81.63%→69.32% accuracy, a ↓12.31% decrease, occurs when the model generates rationales before answers in the text-only QCM→RA setting.The authors attribute this result to misleading or otherwise ineffective rationales rather than exceeding the output-length limit.
- Challenge setup: 90.73 RougeL for rationale generation still yields only 78.57% answer-inference accuracy, below the 81.63% QCM→A result.Thus, generated rationales do not automatically improve answer accuracy.
- Misleading by Hallucinated Rationales: 56% of sampled error cases contain hallucinated rationales that mislead answer inference when vision content is unavailable.The analysis links these mistakes to the lack of necessary visual context for effective multimodal reasoning.
- Multimodality Contributes to Effective Rationales: Adding image captions produces only a marginal ↑0.80% gain, whereas fusing ViT-extracted vision features with language representations raises rationale-generation RougeL to 93.46%.The vision features are fused before decoder input in the two-stage framework.
- Multimodality Contributes to Effective Rationales: 60.7% of hallucination mistakes are corrected with vision features, supporting the choice of the two-stage method for Multimodal-CoT.The authors report that effective rationales contribute to more accurate answer inference and that the two-stage method outperforms one-stage methods.
4 Multimodal-CoT
Multimodal-CoT separates rationale generation from answer inference so multimodal rationales can support final predictions. Its architecture fuses language and vision representations before decoding either rationales or answers.
- Framework Overview: Multimodal-CoT uses two stages: rationale generation followed by answer inference, with both stages sharing the same model structure but differing in inputs and outputs.The first stage generates rationales from language and vision inputs; the second appends the rationale to the original language input while retaining the vision input.
- Framework Overview: The rationale-generation stage takes language and vision inputs and learns to produce a rationale R from the multimodal input X.Language input may concatenate the question, context, and answer options, while vision input represents the associated image.
- Framework Overview: The answer-inference stage appends the generated rationale to the original language input and uses the updated language input with the original vision input to infer the final answer.During inference, rationales generated by the first independently trained model are supplied to the second-stage answer-inference model.
- Model Architecture: The model encodes text with a Transformer and image patches with a vision extractor before correlating text tokens with image patches through single-head attention.The language and vision representations are then combined through a gated fusion mechanism before decoding.
- Model Architecture: Gated fusion combines language and attended vision representations, and the fused output is fed to a Transformer decoder to predict either the rationale or the answer.The same multimodal architecture supports target text generation in both framework stages, while blank image features can represent questions without associated images.
5 Experiments
The experiments evaluate Multimodal-CoT on ScienceQA and A-OKVQA using multiple baseline categories and compact T5-based models. Results show strong benchmark performance, with both multimodal inputs and the two-stage design contributing to gains.
- Datasets: The evaluation uses ScienceQA and A-OKVQA, multimodal reasoning benchmarks with annotated reasoning chains.
- Implementation: The framework uses T5 encoder-decoder models with 200M and 700M parameter settings, initialized with FLAN-Alpaca.
- Baselines: The baseline comparison covers VQA models, text-only language models, few-shot LLMs, and fine-tuned large vision-language models.
- Main results: 86.54%→90.45%: Multimodal-CoT_Large improves over the prior best published ScienceQA result.
- Ablation: Ablations show that both integrating vision features and using the two-stage framework contribute to overall performance.
- Findings: Multimodal-CoT mitigates hallucination and improves convergence according to the reported analysis.
6 Analysis
The analysis examines convergence, rationale supervision, backbone and vision-feature choices, out-of-domain generalization, and error types. Across these studies, Multimodal-CoT remains effective, while commonsense reasoning is the dominant error source.
- Convergence: Two-stage methods achieve higher accuracy early in training than one-stage baselines, while vision features are important for sustained performance.
- Rationale supervision: Generated rationales achieve comparable performance to human-annotated rationales and outperform direct answer prompting by baseline models.
- Model and feature choices: The approach is generally effective with different backbone language models and performs best comparatively with ViT vision features.
- Generalization: Multimodal-CoT generalizes to MMMU without further training and outperforms various larger models around 8B parameters.
- Error analysis: 80% of analyzed errors are commonsense mistakes, followed by 14% logical mistakes and 6% cases with empty or correct CoT but incorrect answers.
- Limitations and future work: Future improvements include more informative visual features, stronger vision-language interaction, commonsense knowledge, and filtering relevant rationales.
7 Conclusion
The paper concludes that Multimodal-CoT combines language and vision in a two-stage rationale-and-answer framework. It reports strong ScienceQA performance, reduced hallucination, faster convergence, and unresolved error sources for future work.
- Conclusion: Multimodal-CoT separates rationale generation from answer inference so answers can use multimodally grounded rationales.
- Conclusion: The model achieves state-of-the-art ScienceQA performance with fewer than 1 billion parameters.
- Conclusion: The analysis identifies hallucination mitigation and enhanced convergence as advantages of Multimodal-CoT.
- Future work: The paper highlights improved vision features, commonsense knowledge, and filtering mechanisms as directions for future CoT reasoning.
A.2 Two-Stage Training Performance with Different Sizes of LMs
This appendix studies how model size and multimodal reasoning paradigms affect rationale-based inference. Scaling improves the text-only baseline, but vision features remain more effective, while the paper uses feature-level multimodal CoT.
- Different LM sizes: Larger language models substantially improve baseline accuracy without vision features, but performance remains considerably below the vision-feature setting.
- Multimodal-CoT paradigms: Caption-based reasoning converts images to captions before answer inference, making performance dependent on image-caption quality.
- Multimodal-CoT paradigms: CoT-based reasoning changes the format from <question →answer> to <question →rationale →answer>.
- Multimodal-CoT paradigms: Multimodal-CoT uses <question + image →rationale →answer> and feature-level interactions between vision and language inputs.
- Vision features: ViT, CLIP, DETR, and ResNet are compared as vision features, with ViT selected by default after achieving relatively better performance.
B.2 Datasets
The evaluation uses ScienceQA and A-OKVQA, two multimodal reasoning benchmarks with annotated reasoning information. ScienceQA emphasizes diverse science questions, while A-OKVQA requires commonsense and world knowledge.
- ScienceQA: ScienceQA contains 21k multimodal multiple-choice questions spanning 3 subjects, 26 topics, 127 categories, and 379 skills.Its training, validation, and test splits contain 12k, 4k, and 4k questions, respectively.
- A-OKVQA: A-OKVQA is a knowledge-based visual question answering benchmark with 25k questions requiring broad commonsense and world knowledge.Each question includes a rationale explaining why an answer is correct, and the train/validation/test split is 17k/1k/6k.
- Evaluation: The model is evaluated on ScienceQA’s test set and A-OKVQA’s validation set because A-OKVQA’s test set is hidden.
B.3 Implementation Details of Multimodal-CoT
The implementation uses T5 encoder-decoder models to generate multimodal reasoning chains, with vision features incorporated into the framework. It also explores pseudo-rationales as an alternative training target.
- Model architecture: The framework adopts T5 encoder-decoder backbones under Base (200M) and large (700M) settings for multimodal reasoning-chain generation.FLAN-Alpaca initializes the model weights.
- Model architecture: The method uses vision features alongside language inputs to support Multimodal-CoT reasoning.
- Training targets: Generated pseudo-rationales can serve as training targets instead of human-annotated reasoning chains.
- Rationale examples: Figure 10 presents examples of generated rationales.
C.2 Detailed Results of Multimodal-CoT on Different Backbone Models
Multimodal-CoT remains effective when the underlying language model backbone is changed. The section reports detailed results across different backbone models.
- Backbone generality: Multimodal-CoT is generally effective across widely used backbone language models.The approach is tested with other variants of different types, with detailed results reported in Table 13.
- Backbone generality: Table 13 provides detailed Multimodal-CoT results for different backbone models.
D Examples of Case Studies
The case-study analysis categorizes incorrect answers and identifies commonsense and logical reasoning failures, along with cases where the rationale does not determine the final answer. It also points to improvements in visual understanding, commonsense knowledge, and rationale filtering.
- Error analysis: 80% of incorrect cases are commonsense mistakes involving map interpretation, image counting, or alphabet use.
- Error analysis: 14% of incorrect cases are logical mistakes involving comparisons or contradictions in the reasoning process.
- Error analysis: 6% of incorrect cases have an empty or correct CoT despite an incorrect final answer.In these cases, the CoT may not influence the final answer.
- Future directions: Future improvements include richer visual features, stronger vision-language interaction, commonsense knowledge, and filtering to retain relevant CoTs.These directions target map and numerical-counting comprehension and the use of relevant rationales for answer inference.
- Case studies: The examples include a speed comparison where the rationale correctly identifies the slower car from distance traveled over equal times.