DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

• Existing multimodal CoT methods require costly rationale annotation and are limited in learning flexibility, unseen reasoning paths, cross-subject transfer, and rationale interpretability.MM-CoT rationales do not benefit zero-shot prompting, while GPT-3-generated rationales do not improve fine-tuned models.Source: §1 · pp. 1–2
• DDCoT uses negative-space prompting to surface uncertainty, then assigns reasoning to LLMs and recognition to visual models before joint reasoning to reduce hallucinations.The approach is motivated by rationale sensitivity in zero-shot prompting and intensified hallucinations from interleaved multimodal inputs.Source: §1 · pp. 2–3
• Generated rationales guide LMs’ chain of thought and attention to multimodal inputs in zero-shot and fine-tuning settings, with DLP and RCVE supporting fine-tuning.For fine-tuning, deep-layer prompting and rational-compressed visual embedding filter, encode, and jointly infer over interleaved multimodal inputs.Source: §1 · pp. 2–3
• On image-context questions, DDCoT improves GPT-3 by +2.53% and UnifiedQA by +8.23% over state-of-the-art LMs while exhibiting generalizability and explainability.Source: §1 · p. 3

CoT Reasoning of LLMs

• Zero-shot [25] and few-shot [60] [44] CoT prompts improve LLM reasoning, while related methods refine example selection and reasoning pipelines.This work extends CoT reasoning to multimodality and its inherent complications.Source: §2 · p. 3

Transferring Specialized Reasoning Skills to Small Models

• Prior work distills LLM CoT capabilities into smaller models [35] [13], whereas this work uses LLM-generated rationales to guide image understanding.The rationales are used for multimodal reasoning rather than generated at inference as in the cited distillation work.Source: §2 · p. 3

Cross-modal CoT Reasoning

• ScienceQA [31] introduced annotated multimodal rationales, and MM-CoT [71] uses a two-stage framework whose rationales benefit either zero-shot or fine-tuning learning, but not both.ScienceQA work prompted GPT-3 and fine-tuned UnifiedQA to generate rationales and answers simultaneously; MM-CoT first learns rationales from ground-truth annotations, then generates the final answer using available information.Source: §2 · p. 3

Integrate Visual Modality to Language Models

• Visual integration methods align image features with language using additional training data [16] [26] [67] [28] [69] [12] or convert off-the-shelf vision-model outputs into text [33] [62] [47].Unlike approaches that integrate multimodal inputs once, this work identifies uncertainty in rationale generation and complements visual information step by step.Source: §2 · p. 3

3.1.1 Two-step Reasoning Process: Multimodal Rationales Generation and Utilization

• Adding rationales alone did not improve multimodal accuracy: answer-only prompting achieved 74.04%, compared with 75.17% for rationale-and-answer generation.The authors suggest LLMs struggle to jointly reason over questions and dense image information, motivating a two-step process that generates rationales and then uses them as input.Source: §3.1.1 · p. 4

3.1.2 Roles of Rationales Differ in Zero-shot and Fine-tuning Learning

• Rationale use differs by learning setting: zero-shot models follow input rationales, while fine-tuned models need both correct rationales and comprehensive prior knowledge.Fine-tuned models show greater error tolerance but are more vulnerable to knowledge deficiencies, which the authors attribute to catastrophic forgetting.Source: §3.1.2 · pp. 4–5

3.1.3 Challenge Lies in LLMs with Multimodality: Hallucinations Intensified

• General rationale generation is challenged by unreliable or incomplete rationales and multimodal hallucinations: manual-annotation-based methods risk misleading models, while naive prompting yields visually irrelevant rationales or fabricated image details.The authors report that training on manual rationales can fail to provide knowledge for out-of-distribution questions; interleaving captions and questions can prompt invented visual information.Source: §3.1.3 · p. 5

3.2 Zero-shot DDCoT Prompting for Multimodal Rationale Generation

• DDCoT generates multimodal rationales by decomposing questions, marking visually unanswerable sub-questions as uncertain, obtaining visual complements from a VQA model, and integrating the information through critical joint reasoning.The LLM is prompted to filter potentially invalid supplementary information and may rectify recognition errors during joint reasoning.Source: §3.2 · pp. 6–7

3.3 Utilization of Multimodal Rationales in General Scenarios

• For fine-tuning, deep-layer prompting aligns multimodal inputs across encoder layers, while rationale-compressed visual embeddings filter image features using textual context and rationales.The visual filtering uses cross-attention to update global features and low-rank intermediate vectors to select relevant local inputs.Source: §3.3 · p. 7

4.1 Comparison with State-of-the-Arts in Zero-shot Prompting and Fine-tuning Learning

• DDCoT consistently outperforms prior methods across ScienceQA zero-shot and fine-tuning benchmarks.In zero-shot prompting, it beats the published few-shot method [31] by 2.92% on GPT-3 and 1.84% on ChatGPT, and improves 1.73% on the IMG split over Chameleon [33]. Fine-tuned DDCoT surpasses UnifiedQA by 17.22% on avg split and 21.96% on IMG split, and MM-CoT [71] by an average of 2.43%.Source: Table 1 · p. 8, §4.1 · pp. 8–9
• When one domain is held out during fine-tuning, DDCoT exceeds MM-CoT by 15.5%, 9.6%, and 12.2% across the three divisions.The experiment trains with two visible domains and evaluates the remaining unseen domain, indicating generalization to out-of-distribution data.Source: Table 2 · p. 9, §4.1 · p. 9

4.2 Analysis Effects of Visual Modality, Rationale Generation, and Fine-tuning Components

• Adding captions to the language-only baseline improves IMG split performance by 2.33%, while DLP and RCVE yield 75.16% on IMG, comparable to caption inputs.Direct image input without these components performs poorly; the analysis also reports slight TXT split degradation, potentially due to different IMG and TXT training inputs.Source: Table 2 · p. 9, §4.2 · p. 9
• Negative-space prompting improves rationale-generation results by 2.58% on IMG and 2.06% on average splits, while explicit uncertainty adds 5.15% on IMG and 2.19% on average.Naive image-agnostic rationales provide no IMG benefit and slightly degrade performance; deconstruction and joint reasoning without specialized designs yield only tiny gains.Source: Table 3 · p. 9, §4.2 · p. 9
• Interleaved visual information reduces rationale authenticity by 28.1% versus naive prompting, while duty-distinct design mitigates hallucinations and explicit uncertainty suppresses them further.The human evaluation of hallucinations aligns with the findings from the rationale-component ablation.Source: Table 4 · p. 10, §4.2 · p. 10
• Replacing RCVE with original visual features reduces average and IMG split performance by 1.28% and 3.02%; removing DLP reduces them by 0.49% and 0.99%.DLP assists alignment, while RCVE provides rational-compressed visual embeddings.Source: Table 5 · p. 10, §4.2 · p. 10
• DDCoT rationales significantly surpass prior methods across all human-evaluation criteria despite modest automatic-metric results.Human criteria include relevance, correctness, completeness, coherence, and explainability; automatic metrics measure similarity to annotated ground-truth rationales.Source: Table 6 · p. 10

5 Conclusion

• DDCoT achieves state-of-the-art zero-shot performance and demonstrates significant reasoning ability on ScienceQA using deep-layer prompting and rational-compressed visual embedding.Source: §5 · p. 10
• The experiments support DDCoT’s superiority and generalization ability in both zero-shot learning and fine-tuning.Source: §5 · p. 10
• The authors report no use of personal, data-security, or ethical-hazard information, but caution that training-data biases may persist and outputs require interpretation.Source: §5 · p. 10

Appendix

• Appendix Section A covers insights across GPT models, quantitative ablations of DDCoT and visual components, and fine-tuning hyperparameters.Source: Appendix · p. 15
• Appendix Section A also includes experiments with existing pretrained VLMs and multimodal reasoning models, plus captioning and video question-answering tasks.Source: Appendix · p. 15
• The appendix also outlines human evaluation, detailed prompts, case studies, and limitations.Source: Appendix · p. 15

A Additional Results

• The example asks which of two magnet pairs has the stronger magnetic force, or whether their forces are equal.The options compare Pair 1, Pair 2, and equal force; the context says the pairs do not affect each other and all magnets share the same material.Source: A Additional Results · p. 15
• Both pairs show magnets with the same pole orientation, while the gap is labeled 4 in. for Pair 1 and 2 in. for Pair 2.In each pair, the left magnet has S on its left and N on its right, as does the right magnet.Source: A Additional Results · p. 15
• Figure 6 shows cases in which GPT-3 and ChatGPT have difficulty understanding dense image information.Source: Figure 6 · p. 15

A.1 Insights for Different GPT Models

• Both GPT-3 [3] and ChatGPT [37] struggle to understand dense image information, with difficulties worsening for detailed captions and limiting rationale-free reasoning in practice.Rationale-free reasoning may work in the examples, but practical use is constrained by dense images and the difficulty of obtaining high-quality captions.Source: §A.1 · p. 16
• Correct answers without rationales indicate commonsense knowledge, but incorrect rationales can mislead both GPT-3 and ChatGPT [37].Figure 7 provides additional cases of rationale-sensitive reasoning.Source: §A.1 · p. 16, Figure 7 · p. 16
• GPT-3 and ChatGPT [37] hallucinate image details when captions are insufficient and may still produce information inconsistent with ample captions.Figure 8 illustrates examples of hallucinations during rationale generation.Source: §A.1 · pp. 16, 18, Figure 8 · p. 17
• DDCoT alleviates hallucinations by decomposing questions into recognition tasks and emphasizing uncertainty, using BLIP-2 [26] as a visual question-answering component, but does not eliminate the problem.The authors report better performance and interpretability than direct methods relying on manually crafted rationales, while noting remaining difficulties in some cases.Source: §A.1 · p. 18

A.2 Quantitative ablations on our DDCoT and visual components

• DDCoT prompting and visual components each provide IMG gains, but together yield substantial gains for multimodal reasoning.The study reports that the two components cooperatively facilitate inducing visual information into language models.Source: §A.2 · p. 18
• With answers removed from annotated rationales, the proposed rationale achieves fine-tuning performance comparable to the annotations.This comparison uses the remaining annotation text as input rationales.Source: §A.2 · p. 18
• ScienceQA ground-truth rationales include final predictions, so answers were manually excluded and the remaining text used as fine-tuning input rationales for fair comparison.Source: §A.2 · p. 18

A.3 Hyperparameters for Fine-tuning Components

• Setting Np to 3 and Nr to 16 yields the best performance among the tested fine-tuning component settings.Np is the number of learnable prompt tokens; Nr is the number of low-rank vectors.Source: §A.3 · p. 18
• The Np ablation tested 1, 3, and 5 prompts; more prompts add learnable parameters and guidance, but too many may disrupt visual-feature comprehension and lower performance.Source: §A.3 · p. 18
• The Nr ablation tested 8, 16, and 32 low-rank vectors to vary filtering intensity in Rational-Compressed Visual Embedding.Source: §A.3 · p. 18

A.4 Additional experiments on the effectiveness of our DDCoT with existing pre-trained VLMs and multimodal reasoning models

• DDCoT-generated rationales are compatible with existing pretrained VLMs.The section reports that without correct rationales, pretrained VLMs and multimodal reasoning models have difficulty with complex reasoning tasks.Source: §A.4 · pp. 18–19
• 11.14% improvement is reported based on Flamingo, and 10.96% based on Mini GPT-4, with DDCoT rationales helping models comprehend visual information and reason with rich knowledge.The comparison is reported in Table 9.Source: Table 9 · p. 19, §A.4 · p. 19

A.5 Additional experiments on Captioning and Video Question Answering tasks

• The experiments extend DDCoT to NoCaps captioning and MSVD-QA video question answering, datasets chosen because other existing datasets may not fully exploit its benefits.The experiments are presented in Table 10.Source: §A.5 · p. 19
• 34.94%, 3.15%, and 1.18% were reported for SkipThoughtCS, EmbeddingAverageCS, and GreedyMatchingScore, respectively, by enriching simple captions with LLM-generated sub-answers.The authors note that CIDEr is limiting because it measures similarity to reference captions without considering diversity or high-level semantics.Source: §A.5 · p. 19
• 4.9% improvement over BLIP-2 was achieved on MSVD-QA by decomposing questions, sampling video frames for VQA recognition, and integrating frame information into rationales and answers.The approach combines the LLM’s sequence understanding with visual information returned by the VQA model.Source: §A.5 · p. 19
• Evaluation was limited to 1,000 randomly selected NoCaps images and 1,000 MSVD test videos in a zero-shot setting.Source: §A.5 · p. 19

B Human Evaluation

• Evaluators rate GPT, MM-COT [71], and DDCoT rationales on relevance, correctness, completeness, coherence, and explainability using a 0–5 scale.Scores are averaged for each aspect, with overall scores and ratios relative to the maximum score reported.Source: B Human Evaluation · p. 19
• Questions and rationales are divided into 12 groups, with three evaluators assigned to each group.The evaluation interface includes instructions and an example question (Figure 9).Source: B Human Evaluation · p. 19, Figure 9 · p. 20

C Detailed Prompts

• ChatGPT[37] is the key component of the zero-shot rationale generator, handling the duties of reasoning and recognition and the joint reasoning step.Figure 10 presents the complete zero-shot DDCoT prompts.Source: C Detailed Prompts · pp. 19–20

B Human Evaluation

• Figure 9 shows the human-evaluation interface, including instructions and an example question.The example question contains a question, context, options, and an image.Source: B Human Evaluation · p. 19, Figure 9 · p. 20

C Detailed Prompts

• Figure 10 presents the complete prompts used for zero-shot DDCoT prompting.The prompts cover the duties of reasoning and recognition and the joint reasoning step.Source: C Detailed Prompts · p. 20

D Case Studies

• DDCoT integrates visual features with common knowledge to answer map-related questions and uses image-derived information for more complex reasoning, but can hallucinate on complex image–text interactions.The selected test cases include correct answers as well as erroneous reasoning and answers caused by hallucinations.Source: D Case Studies · p. 20

E Limitations

• The model remains susceptible to hallucinations despite mitigating some multimodal hallucination, leaving suppression methods for future research.The authors point to Figure 13 as an illustration of this risk.Source: E Limitations · p. 20, Figure 13 · p. 24
• The study used no extra image-text pairs to pre-train vision-language alignment, which the authors expect could further improve joint reasoning.Source: E Limitations · p. 20
• Zero-shot LLM rationale generation may inherit cultural, ethical, and other social biases that could adversely affect users.Possible future mitigations include constraints at each prompting stage or more advanced LLMs trained on unbiased resources.Source: E Limitations · p. 20