PaperlayerView exact revision on arXiv

Source-linked AI summary

DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, Sibei Yang

arXiv:2310.16436v2cs.CVcs.CL

TL;DR

Multimodal CoT is constrained by labor-intensive rationale annotation and limited flexibility and generalizability across learning settings. DDCoT addresses these challenges by using negative-space prompting and dividing reasoning and recognition between language and visual models. The resulting rationales improve GPT-3 and UnifiedQA performance on image-context questions and support zero-shot and fine-tuning learning, with reported generalizability and explainability.

  • Problem

    Existing multimodal CoT methods require labor-intensive rationale annotation and have limited flexibility and generalizability across zero-shot prompting and fine-tuning learning.

  • Method

    DDCoT generates multimodal rationales using negative-space prompting, separates reasoning from visual recognition, and uses the rationales to guide multimodal reasoning.

  • Results

    +2.53% for GPT-3 and +8.23% for UnifiedQA on questions with image context, with reported generalizability and explainability.

  • Takeaways & Limitations

    The generated rationales improve language-model reasoning in both zero-shot prompting and fine-tuning learning.

  • Takeaways & Limitations

    The appendix illustrates rationale outputs for gas-particle temperature and solution-concentration questions, constraining the supplied limitation evidence to these examples.

Abstract

from arXiv · show

A long-standing goal of AI systems is to perform complex multimodal reasoning like humans. Recently, large language models (LLMs) have made remarkable strides in such multi-step reasoning on the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. However, the transfer of these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation and the limitations in terms of flexibility, generalizability, and explainability. To evoke CoT reasoning in multimodality, this work first conducts an in-depth analysis of these challenges posed by multimodality and presents two key insights: "keeping critical thinking" and "letting everyone do their jobs" in multimodal CoT reasoning. Furthermore, this study proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process. The rationales generated by DDCoT not only improve the reasoning abilities of both large and small language models in zero-shot prompting and fine-tuning learning, significantly outperforming state-of-the-art methods but also exhibit impressive generalizability and explainability.

1 Introduction

Multimodal CoT methods face annotation, flexibility, and generalizability challenges that limit rationale use across zero-shot and fine-tuning settings. DDCoT addresses these issues by separating reasoning from visual recognition, using negative-space prompting, and applying rationales to both learning settings.

  • Language-model CoT has advanced multi-step reasoning, but most prior work focuses on language rather than multimodal contexts.
  • Existing multimodal CoT methods require labor-intensive rationale annotation and support either zero-shot prompting or fine-tuning learning, not both effectively.
  • DDCoT generates multimodal rationales with language-only LLMs through negative-space prompting and distinct reasoning and recognition responsibilities.
  • DDCoT uses generated rationales to guide multimodal chain-of-thought reasoning in zero-shot prompting and fine-tuning learning.Fine-tuning uses deep-layer prompting and rational-compressed visual embedding to filter, encode, and jointly infer over interleaved multimodal inputs.
  • +2.53% for GPT-3 and +8.23% for UnifiedQA on questions with image context, while the methods also exhibit generalizability and explainability.

2 Related Work

Prior work develops CoT reasoning for language models, smaller models, and vision-language tasks, but multimodal methods differ in how they generate and use rationales. Related approaches include rationale distillation, annotated multimodal CoT, and methods that integrate visual modality into language models.

  • CoT Reasoning of LLMs: Language-model CoT research improves multi-step reasoning through zero-shot, few-shot, example-selection, and reasoning-pipeline prompts.
  • Transferring Specialized Reasoning Skills to Small Models: CoT capabilities can be distilled from large language models into smaller models for specific tasks.
  • Cross-modal CoT Reasoning: ScienceQA introduced multimodal CoT with annotated scientific questions, using GPT-3 for zero-shot rationale-and-answer generation and UnifiedQA for fine-tuning.
  • Cross-modal CoT Reasoning: MM-CoT uses a two-stage framework that learns rationales from ground-truth annotations before generating final answers, with benefits limited to either zero-shot or fine-tuning learning.
  • Integrate Visual Modality to Language Models: Visual-modality integration approaches align image features with linguistic space or use language models’ scheduling ability for multimodal problems.

3 Method

The method analyzes how multimodal rationales function differently in zero-shot and fine-tuning settings, then proposes DDCoT to generate and use more reliable, general rationales. DDCoT separates reasoning from visual recognition, marks uncertainty, and integrates visual complements through joint reasoning.

  • Rationale-Guided Multimodal Reasoning: 74.04% accuracy was achieved when generating only answers versus 75.17% when generating rationales and answers, showing limited benefit from directly prompting LLMs for both.The reported comparison motivates a more structured rationale-generation process.
  • Rationale-Guided Multimodal Reasoning: Rationales improve multimodal reasoning when provided as structured logical chains rather than dense image information alone.The proposed process first generates multimodal rationales and then uses them to guide reasoning.
  • Different Roles Across Learning Settings: Zero-shot models tend to follow input rationales, so misleading claims can cause failures even when the model’s own knowledge would support the correct answer.Fine-tuning models additionally require comprehensive prior knowledge in rationales, while showing greater error tolerance than zero-shot prompting.
  • Challenges in Multimodal Rationale Generation: Existing methods face rationale-correctness, knowledge, and hallucination challenges, especially when interleaved visual and linguistic information is supplied simultaneously.Unimodal rationales can be visually irrelevant, while naive caption-question prompts still produce suboptimal image-related rationales.
  • DDCoT Prompting: DDCoT decomposes questions into sub-questions, uses visual models to answer recognition sub-problems, and asks an LLM to integrate the resulting sub-answers.Its prompts also mark uncertain supplementary information so the LLM can select valid information and apply critical thinking.
  • Rationale Utilization: For utilization, DDCoT combines generated rationales with problem statements for zero-shot prompting and uses rationale-guided visual filtering for fine-tuning.The visual filtering stage introduces low-rank intermediate vectors to capture relevant local visual inputs.

4 Experiment

Experiments on ScienceQA compare DDCoT with prior methods across zero-shot prompting, fine-tuning, generalization, modality, rationale, hallucination, and component analyses. DDCoT consistently improves performance, including on unseen domains and image-context questions.

  • Main Results: DDCoT consistently achieves superior performance to previous methods on zero-shot and fine-tuning benchmarks.The comparison includes published and preprinted state-of-the-art methods.
  • Zero-shot Prompting: 2.92% on GPT-3 and 1.84% on ChatGPT are DDCoT’s zero-shot gains over the published few-shot method UnifiedQA.DDCoT also achieves a 1.73% enhancement on the IMG split compared with Chameleon.
  • Fine-tuning Learning: 17.22% on avg split and 21.96% on IMG split are DDCoT’s gains over the UnifiedQA base model in fine-tuning.Zero-shot-generated multimodal rationales also improve performance by an average of 2.43% over MM-CoT with annotated rationales.
  • Generalization: 15.5%, 9.6%, and 12.2% are DDCoT’s gains over MM-CoT across three unseen-domain divisions.The authors interpret these results as evidence of rationale scalability on out-of-distribution data.
  • Ablation and Human Analysis: Explicit uncertainty improves DDCoT by 5.15% on IMG split and 2.19% on average, while interleaved visual information reduces rationale authenticity by 28.1%.Duty-distinct design mitigates hallucinations; removing rational-compressed visual embedding reduces average and IMG performance by 1.28% and 3.02%.

5 Conclusion

The conclusion presents DDCoT as a multimodal reasoning approach that achieves strong zero-shot and fine-tuning results with generalization on ScienceQA. It also notes that pretrained language models may retain biases and that their outputs require interpretation.

  • Conclusion: DDCoT combines multimodal prompting with deep-layer prompting and rational-compressed visual embedding for language-model reasoning.The paper reports significant reasoning ability on the ScienceQA benchmark.
  • Conclusion: DDCoT demonstrates state-of-the-art zero-shot performance and superiority and generalization ability in both zero-shot learning and fine-tuning.These claims are made in the paper’s concluding experimental summary.
  • Ethical Statement: Large language models may introduce biases from their training datasets, and current technology may not eliminate them completely.The authors advise users to interpret model outputs rather than treat them as infallible.

A Additional Results

The additional-results section includes a magnetic-force question example and a figure documenting cases where GPT-3 and ChatGPT struggle with dense image information.

  • Additional Results: A supplementary example asks which statement about the magnetic force between each pair of magnets is true.The supplied passage provides the question but not its answer or associated analysis.
  • Additional Results: Figure 6 presents cases where GPT-3 and ChatGPT have difficulty understanding dense image information.The passage identifies the figure’s subject but does not state a quantitative comparison or outcome.

A.1 Insights for Different GPT Models

GPT-3 and ChatGPT can be highly sensitive to rationale quality: misleading rationales distort answers, while hallucinations arise when captions underspecify image details. DDCoT reduces these hallucinations by separating recognition from reasoning and emphasizing uncertainty.

  • Both models struggle to understand image information from sparse captions, with difficulties becoming more pronounced for dense captions.
  • GPT-3 and ChatGPT can produce correct answers without rationales, indicating commonsense knowledge, but incorrect rationales can mislead both models.
  • When captions lack sufficient information, GPT-3 and ChatGPT tend to imagine image details while generating rationales.
  • DDCoT alleviates hallucinations by decomposing questions into recognition tasks and emphasizing uncertainty in image-related aspects.
  • Hallucinations remain difficult to eliminate entirely, with some problematic cases persisting despite the proposed mitigation.

A.2 Quantitative ablations on our DDCoT and visual components

Ablation studies show that DDCoT prompting and visual components work cooperatively, while manually removing answer information from annotated rationales leaves DDCoT performance comparable to those rationales.

  • DDCoT prompting and visual components individually provide gains on IMG improvement, but their combination yields substantial gains.
  • After answers were removed from annotated ScienceQA rationales, DDCoT achieved fine-tuning performance comparable to the annotated rationales.
  • The ablation varies Np, the number of learnable prompt tokens, and Nr, the number of low-rank vectors used for visual-embedding filtering.
  • Selecting Np=3 and Nr=16 produced the best performance in the reported hyperparameter experiments.

A.4 Additional experiments on the effectiveness of our DDCoT with existing pre-trained VLMs and multimodal reasoning models

DDCoT-generated rationales are compatible with existing pretrained vision-language and multimodal reasoning models. They improve complex visual reasoning performance for Flamingo and MiniGPT-4.

  • Without correct rationales, existing pretrained VLMs and multimodal reasoning models have difficulty with complex reasoning tasks.
  • DDCoT-generated rationales improve performance by 11.14% for Flamingo and 10.96% for MiniGPT-4.

A.5 Additional experiments on Captioning and Video Question Answering tasks

Additional experiments extend DDCoT beyond ScienceQA to image captioning and video question answering. The approach improves semantic caption metrics and outperforms BLIP-2 on MSVD-QA, but evaluation uses only 1,000 images and 1,000 videos.

  • The authors evaluate DDCoT on NoCaps captioning and MSVD-QA video question answering in zero-shot settings.
  • Captioning improves SkipThoughtCS by 34.94%, EmbeddingAverageCS by 3.15%, and GreedyMatchingScore by 1.18%.
  • Video question answering improves by 4.9% over BLIP-2 by combining sampled-frame VQA recognition with LLM sequence understanding.
  • The additional experiments randomly evaluate only 1,000 NoCaps images and 1,000 MSVD test videos.

B Human Evaluation

The paper evaluates rationale quality across five human-judged dimensions and illustrates how DDCoT integrates visual features with common knowledge in multimodal reasoning.

  • Human evaluation: Evaluators rate GPT, MM-CoT, and DDCoT rationales for relevance, correctness, completeness, coherence, and explainability.Each sample includes a question, context, options, and image.
  • Prompting procedure: Zero-shot DDCoT prompting uses ChatGPT to break reasoning into duty-distinct recognition and reasoning steps before joint reasoning.Figure 10 presents the complete prompts for this process.
  • Human evaluation: The human-evaluation interface provides instructions and an example question for rating generated rationales.
  • Qualitative examples: Map-related examples show DDCoT combining visual features, such as highlighted-area shape, with common knowledge to reach correct answers.The examples are presented as illustrations of the rationale-generation process.

E Limitations

The supplied material highlights remaining hallucination and bias risks while illustrating DDCoT’s rationale-generation process through recognition sub-questions, supplementary answers, and worked examples.

  • Limitations: DDCoT reduces some multimodal hallucinations, but hallucinations remain possible and require further suppression research.The authors also state that additional image-text pre-training is expected to improve vision-language alignment for joint reasoning.
  • Limitations: Zero-shot rationale generation may inherit cultural, ethical, and other social biases from the underlying LLM.The paper suggests future constraints at prompting stages or more advanced LLMs trained on unbiased resources.
  • Rationale generation: DDCoT decomposes visual questions into recognition-oriented sub-questions whose answers are supplied as supplementary information for subsequent reasoning.Examples ask about highlighted-area shape, geography, and surrounding bodies of water before generating a rationale.
  • Worked examples: Worked examples show rationales combining supplementary information with domain knowledge to infer answers about geography, temperature, concentration, precipitation, and genetics.Some examples also contain uncertainty or contradictory intermediate answers, underscoring the need to select valid information.
Loading 2310.16436v2…