Multimodal Chain-of-Thought Reasoning in Language Models

• Earlier CoT studies largely focused on language, so the paper advocates extending multi-step rationale generation and answer inference to multimodal inputs.This work focuses on vision and language.Source: §1 · p. 2
• Because caption-based prompting loses visual information and large LLMs are costly to deploy, the work fine-tunes smaller models, whose hallucinated rationales remain a challenge.Fine-tuning allows model architectures to be adjusted to incorporate multimodal features; the cited hallucination concern applies to language models under 100 billion parameters.Source: §1 · p. 2
• Multimodal-CoT separates rationale generation from answer inference, using fused vision and language representations on ScienceQA and A-OKVQA.The benchmarks have annotated reasoning chains, and the generated rationales are intended to facilitate final-answer inference.Source: §1 · p. 2
• The method achieves state-of-the-art performance on ScienceQA and is reported to mitigate hallucination and boost convergence.Source: §1 · p. 2
• The authors describe this as, to their knowledge, the first study of CoT reasoning across modalities in scientific peer-reviewed literature, focusing on models under 1 billion parameters without LLM outputs.The paper defines its small models as having less than 1 billion parameters.Source: §1 · p. 2, Table 1 · p. 3
• The authors report that vision features alleviate failures of naive CoT, with the approach effective across tasks and backbone models.Source: §1 · p. 2

2.1 CoT Reasoning with LLMs

• CoT prompting uses zero-shot cues or few-shot reasoning demonstrations, with few-shot methods often outperforming zero-shot methods when demonstrations are effective.Zero-Shot-CoT adds a cue such as “Let’s think step by step”; Few-Shot-CoT conditions inference on question-and-reasoning examples, which may be manually written or automatically generated.Source: §2.1 · p. 3
• Demonstration optimization addresses quality and selection: Auto-CoT diversifies questions, while other approaches select complex examples or learn example choices.Different annotators’ demonstrations can yield accuracy disparities (Wei et al., 2022b); semantic retrieval can degrade when reasoning chains contain mistakes (Rubin et al., 2022; Zhang et al., 2023d). Auto-CoT clusters questions and generates chains for representative examples (Zhang et al., 2023d); alternatives use reasoning complexity (Fu et al., 2022) or reward-trained selection (Lu et al., 2022b).Source: §2.1 · p. 3
• Reasoning-chain methods decompose problems, represent reasoning as executable programs, or vote across multiple sampled reasoning paths.Least-to-most prompting solves subproblems sequentially (Zhou et al., 2022), and diverse decomposition structures address subquestions (Khot et al., 2022); program-of-thoughts executes generated programs (Chen et al., 2022). Self-consistency takes a majority over sampled final answers (Wang et al., 2022b), while other methods add input randomness for more diverse voting outputs (Wang et al., 2022c; Li et al., 2022c).Source: §2.1 · p. 4

2.2 Eliciting CoT Reasoning by Fine-Tuning Models

• Fine-tuning work uses annotated CoT data, teacher-student distillation, and context-aware prompt synthesis.Lu et al. (2022a) observed a dramatic decline when generating reasoning before the answer, instead using CoT as a post-answer explanation; Magister et al. (2022) and Ho et al. (2022) distilled teacher-generated chains, while Wang et al. (2022a) synthesized prompts conditioned on current-step context.Source: §2.2 · p. 4
• Models under 100 billion parameters tend to produce illogical CoT, making effective reasoning harder for 1B models than direct answering and harder still with multimodal inputs.The section identifies this as a challenge for training small models to reason over multimodal inputs.Source: §2.2 · p. 4

3.1 Towards the Role of CoT

• 81.63%→69.32% was a 12.31% accuracy decrease when the text-only model generated rationales before answers, despite outputs staying below 400 tokens and the 512-token limit.The finding suggests rationales may harm answer inference for reasons beyond exceeding the model’s maximum output length.Source: §3.1 · p. 5

3.3 Multimodality Contributes to Effective Rationales

• 93.46% RougeL for rationale generation and 85.31% accuracy for answer inference were achieved with vision features in the two-stage setting.Captions alone yielded marginal performance gains of 0.80%; with effective rationales, 60.7% of hallucination mistakes were corrected, and the authors chose the two-stage approach for Multimodal-CoT.Source: Table 3 · p. 5, §3.3 · p. 6

3.2 Misleading by Hallucinated Rationales

• The two-stage text-only baseline scored 90.73 RougeL for rationale generation but only 78.57% accuracy for answer inference, below the QCM→A variant’s 81.63%.This indicates that generated rationales did not improve answer accuracy over direct answer prediction.Source: Table 3 · p. 5, §3.2 · p. 5
• 56% of sampled error cases involved hallucinated rationales, which misled answer inference when the model lacked vision content.The analysis sampled 50 error cases; Figure 2 illustrates a mistaken rationale about the south poles of two magnets.Source: §3.2 · p. 5

4.1 Framework Overview

• Rationale generation uses language and vision inputs to produce R; answer inference appends R to the original language input and combines it with the original image to predict A.The two stages share the same model structure but differ in their inputs and outputs.Source: Figure 4 · p. 6
• The two stages train separate models with the same architecture on annotated rationale-generation and answer-inference examples.At inference, the first model generates test-set rationales that are passed to the second model for answer inference.Source: §4.1 · p. 7

4.2 Model Architecture

• The shared Transformer architecture encodes text and projected image patches, aligns them with single-head attention, fuses them through a gate, and decodes the target.Frozen vision extractors such as ViT provide patch features; image-free inputs use all-zero blank features.Source: §4.2 · pp. 7–8

5.1 Dataset

• ScienceQA has 21k multimodal questions with annotated lectures and explanations, while A-OKVQA has 25k knowledge-based questions; experiments use A-OKVQA multiple choice for consistency.Source: §5.1 · p. 8

5.2 Implementation

• The framework uses T5 Base (200M) and large (700M), initializes weights with FLAN-Alpaca, and extracts vision features using frozen ViT-large.Models are fine-tuned for up to 20 epochs with a learning rate of 5e-5.Source: §5.2 · p. 8
• Comparisons cover VQA classifiers, text-only and few-shot language models, and fine-tuned large vision-language models.Source: §5.2 · p. 8

5.3 Main Results

• 86.54%→90.45%: Multimodal-CoTLarge substantially exceeds the prior best published ScienceQA result, with Table 4 presenting the main benchmark comparison.Source: Table 4 · p. 9, §5.3 · p. 9
• The authors say A-OKVQA results further support Multimodal-CoT’s efficacy; several compared multimodal models were concurrent works, and the method is described as orthogonal to them.Source: §5.3 · p. 9, Table 5 · p. 9
• Ablations attribute performance contributions to both vision-feature integration and the two-stage design, while analysis reports reduced hallucination and improved convergence.Source: §5.3 · p. 9

6.1 Multimodality Boosts Convergence

• Two-stage Multimodal-CoT with vision features yields better answer accuracy over training than one-stage baselines and a two-stage baseline without vision features.The two-stage methods start with relatively higher accuracy, but without vision their later performance suffers from low-quality rationales.Source: §6.1 · p. 10, Figure 5 · p. 10

6.2 When Multimodal-CoT Meets Large Models

• Generated pseudo-rationales achieve comparable performance to human-annotated rationales and outperform direct QCM→A prompting, enabling training without human-annotated reasoning chains.InstructBLIP generated rationales for image-paired questions, while ChatGPT generated rationales for questions without images.Source: §6.2 · pp. 10–11, Table 7 · p. 11

6.3 Effectiveness Across Backbones

• Multimodal-CoT is generally effective with alternative backbone language models.The analysis tests other LM variants from different types.Source: §6.3 · p. 11, Table 8 · p. 11

6.4 Using Different Vision Features

• ViT achieves relatively better performance than the compared vision features, so the paper uses ViT by default.The comparison includes ViT, CLIP, and DETR patch-like features and ResNet features repeated to match text-sequence length.Source: §6.4 · p. 11, Table 9 · p. 11

6.5 Alignment Strategies for Multimodal Interaction

• Both the paper’s alignment strategy and BLIP’s image-grounded text encoder perform better than direct answering.The image-grounded strategy injects visual information through an additional cross-attention layer in each text-encoder transformer block (Li et al., 2022b).Source: §6.5 · p. 11, Table 10 · p. 12

6.6 Generalization to Other Multimodal Reasoning Benchmarks

• Without further training on MMMU, Multimodal-CoT generalizes effectively and outperforms various larger models around 8B.Source: §6.6 · p. 12, Table 11 · p. 12

6.7 Error Analysis

• Among 50 incorrect-answer samples, commonsense mistakes account for 80% of errors, logical mistakes 14%, and cases with empty or correct CoT 6%.The authors suggest improving visual features and language–vision interaction for maps and counting, adding commonsense knowledge, and filtering irrelevant CoTs.Source: §6.7 · p. 12

7 Conclusion

• With fewer than 1 billion parameters, Multimodal-CoT achieves state-of-the-art performance on ScienceQA.Source: §7 · p. 13
• Multimodal-CoT uses a two-stage framework that separates multimodal rationale generation from answer inference.Answer inference can leverage rationales generated from language and vision information.Source: §7 · p. 13
• The analysis finds that Multimodal-CoT mitigates hallucination in reasoning.Source: §7 · p. 13
• The analysis also finds that Multimodal-CoT enhances convergence speed.Source: §7 · p. 13
• Future work could improve CoT reasoning with more effective vision features, injected commonsense knowledge, and filtering mechanisms.Source: §7 · p. 13

A.1 Additional Examples of Misleading through Hallucinated Rationales

• Case studies observed that the baseline model tends to generate hallucinated rationales, and the appendix presents additional examples.Source: §A.1 · p. 19
• Figure 7 contrasts the baseline without vision features and Multimodal-CoT with vision features in rationale generation and answer prediction.The figure's upper part presents problem details, and its lower part shows baseline and method outputs.Source: Figure 7 · p. 19

A.2 Two-Stage Training Performance with Different Sizes of LMs

• Larger LMs significantly improve baseline accuracy without vision, but still fall considerably short of using vision features, supporting Multimodal-CoT across LM sizes.Source: Figure 8 · p. 20, §A.2 · p. 20

A.3 Discussion of the Possible Paradigms to Achieve Multimodal-CoT

• Caption-based reasoning conveys visual information as text by concatenating an image caption with the question, but its effectiveness depends on caption quality.Captioning and transfer errors can affect answer inference.Source: §A.3 · p. 20
• CoT-based reasoning changes question-to-answer prompting into question-to-rationale-to-answer steps, a nomenclature adopted in prior literature (Huang & Chang, 2022; Zhang et al., 2023d; Lu et al., 2022c).Source: §A.3 · p. 20
• The paper uses a multimodal CoT framework in which question and image inform rationale generation before answer inference, enabling feature-level vision-language interactions.Figure 9 contrasts the paradigms for achieving Multimodal-CoT.Source: §A.3 · p. 20, Figure 9 · p. 21
• The authors report that Multimodal-CoT mitigates hallucination, enhances convergence, and performs strongly on benchmark datasets, while its lightweight design suits resource constraints and avoids potential paywalls.Source: §A.3 · p. 20

B.1 Details of Vision Features

• The compared vision features use ViT vit_large_patch32_384, CLIP RN101, DETR detr_resnet101_dc5, and averaged pooled features from a pretrained ResNet50 CNN.The cited model sources are ViT (Dosovitskiy et al., 2021b), CLIP (Radford et al., 2021), DETR (Carion et al., 2020), and ResNet (He et al., 2016).Source: §B.1 · p. 21
• Table 12 presents feature dimensions after VisionExtractor; ResNet-50 pooled features are repeated to the text-sequence length to imitate patch-like features.Each repeated patch is the same pooled image feature.Source: Table 12 · p. 21, §B.1 · p. 21

B.2 Datasets

• ScienceQA contains 21k multimodal multiple-choice questions spanning 3 subjects, 26 topics, 127 categories, and 379 skills.It includes annotated lectures and explanations, with 12k/4k/4k questions in the train/validation/test splits, respectively.Source: §B.2 · p. 21
• A-OKVQA has 25k knowledge-based visual questions requiring commonsense and world knowledge, with rationales and 17k/1k/6k train/validation/test questions.Its rationales explain why an answer is correct based on necessary facts or knowledge.Source: §B.2 · pp. 21–22
• Evaluation uses the ScienceQA test set and the A-OKVQA validation set because its test set is hidden.The method is evaluated on both benchmark datasets.Source: §B.2 · pp. 21–22

B.3 Implementation Details of Multimodal-CoT

• Multimodal-CoT uses T5 encoder-decoder models at Base (200M) and large (700M) scales, initialized with FLAN-Alpaca weights.The paper notes that it will also evaluate UnifiedQA and FLAN-T5 as backbones in Section 6.1.Source: §B.3 · p. 22
• Image features come from a frozen ViT-large encoder, and InstructBLIP-generated captions are appended to context because captions slightly improve performance.Captions are appended following Lu et al. (2022a).Source: §B.3 · p. 22
• Models are fine-tuned for up to 20 epochs using a learning rate selected from 5e-5 and 8e-5, with batch size 8.Source: §B.3 · p. 22
• Rationale generation and answer inference use maximum input sequence lengths of 512 and 64, respectively, and experiments run on 8 NVIDIA Tesla V100 32G GPUs.Source: §B.3 · p. 22

C.1 Examples of Rationale Generation with Large Models

• To explore avoiding datasets with human-annotated rationales, the authors replace human rationale targets with rationales generated by large models during first-stage training.The benchmark datasets’ human annotations are the original targets.Source: §C.1 · p. 22
• For zero-shot rationale generation, questions with images go to InstructBLIP and questions without images go to ChatGPT; the resulting pseudo-rationales become training targets.The examples are identified in Figure 10a and Figure 10b, respectively.Source: §C.1 · p. 22

C.2 Detailed Results of Multimodal-CoT on Different Backbone Models

• Multimodal-CoT is generally effective across widely used backbone models.The authors evaluate different backbone variants; Table 13 presents the detailed results.Source: §C.2 · p. 23, Table 13 · p. 23

D Examples of Case Studies

• Among errors, commonsense mistakes were most prevalent at 80%; logical mistakes made up 14%, while 6% involved incorrect answers despite empty or correct CoT.Source: D Examples of Case Studies · p. 23
• Commonsense errors involved interpreting maps, counting objects in images, or using the alphabet.The authors suggest improving visual features and language–vision interaction to support map comprehension and numerical counting, and incorporating commonsense knowledge.Source: D Examples of Case Studies · p. 23, Figure 12 · p. 24
• Logical mistakes constituted 14% of errors and included faulty comparisons and contradictions in reasoning.Examples are shown in Figure 13.Source: D Examples of Case Studies · p. 23, Figure 13 · p. 25
• In 6% of errors, the answer was incorrect despite an empty or correct CoT, which may not have influenced the final answer.The authors propose filtering CoTs so only relevant ones are used to infer answers.Source: D Examples of Case Studies · p. 23, Figure 14 · p. 25