PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Chain-of-Thought Prompting for Large Multimodal Models

Chancharik Mitra, Brandon Huang, Trevor Darrell, Roei Herzig

arXiv:2311.17076v3cs.CVcs.AIcs.CLcs.LG

TL;DR

LMMs struggle with compositional visual reasoning, while scene-graph supervision is costly and fine-tuning can cause forgetting. CCoT addresses this with zero-shot scene-graph prompting, improving compositional and general multimodal benchmark performance without fine-tuning or annotated SGs.

  • Problem

    LMMs struggle to capture object attributes and relationships, while scene-graph annotations are costly and SG-based fine-tuning can cause forgetting.

  • Method

    CCoT first generates a scene graph from the image and task prompt, then uses that graph with both inputs to produce a response.

  • Results

    CCoT improves several LMMs on compositional benchmarks and general multimodal benchmarks without fine-tuning or annotated ground-truth SGs.

  • Takeaways & Limitations

    Generated scene graphs provide a zero-shot, structured reasoning representation that broadens LMM compositional and multimodal reasoning across tasks.

  • Takeaways & Limitations

    CCoT is limited by current LMM context lengths and is less useful for language-dominant tasks such as document understanding.

Abstract

from arXiv · show

The combination of strong visual backbones and Large Language Model (LLM) reasoning has led to Large Multimodal Models (LMMs) becoming the current standard for a wide range of vision and language (VL) tasks. However, recent research has shown that even the most advanced LMMs still struggle to capture aspects of compositional visual reasoning, such as attributes and relationships between objects. One solution is to utilize scene graphs (SGs)--a formalization of objects and their relations and attributes that has been extensively used as a bridge between the visual and textual domains. Yet, scene graph data requires scene graph annotations, which are expensive to collect and thus not easily scalable. Moreover, finetuning an LMM based on SG data can lead to catastrophic forgetting of the pretraining objective. To overcome this, inspired by chain-of-thought methods, we propose Compositional Chain-of-Thought (CCoT), a novel zero-shot Chain-of-Thought prompting method that utilizes SG representations in order to extract compositional knowledge from an LMM. Specifically, we first generate an SG using the LMM, and then use that SG in the prompt to produce a response. Through extensive experiments, we find that the proposed CCoT approach not only improves LMM performance on several vision and language VL compositional benchmarks but also improves the performance of several popular LMMs on general multimodal benchmarks, without the need for fine-tuning or annotated ground-truth SGs. Code: https://github.com/chancharikmitra/CCoT

1. Introduction

CCoT addresses LMMs’ difficulty representing attributes and relationships by using generated scene graphs in a zero-shot, two-step prompting process. The method avoids annotated SG data and fine-tuning while improving performance across compositional and general multimodal benchmarks.

  • Motivation: LMMs often treat images as bags of objects, making attributes and relationships between objects difficult to capture.Scene graphs represent objects together with their important characteristics and relationships.
  • Motivation: Scene graph annotations are costly and scarce, while SG-based fine-tuning can cause forgetting of pretrained objectives.Visual Genome, for example, contains approximately 100K image-SG pairs.
  • Approach: CCoT is a zero-shot Chain-of-Thought method that first generates a scene graph and then uses it with the image and task prompt to produce a response.The generated graph supplies compositional information as an intermediate reasoning representation.
  • Approach: The method requires neither task-specific fine-tuning nor annotated SG data and is designed to work across different LMM architectures.Generated scene graphs can describe arbitrary visual scenes, supporting varied vision-language tasks.
  • Results: CCoT improves four LMMs on compositional benchmarks including Winoground and WHOOPS!, as well as general benchmarks including SEEDBench, MMBench, and LLaVA-Bench-in-the-Wild.The reported models are LLaVA-1.5, InstructBLIP, SPHINX, and GPT-4V.

2. Related Work

Prior multimodal reasoning methods use visual features, captions, or code, but differ in how directly and flexibly they represent visual structure. CCoT instead uses generated scene graphs as an inference-time reasoning step for compositional and broader multimodal reasoning.

  • Multimodal Models: LMMs combine language-model reasoning with visual representations, but visual instruction tuning typically depends on high-quality training data.
  • Visual Programmatic Models: Visual Programmatic Models use code generation as a proxy for visual reasoning but remain limited by the APIs or models available through their context.CCoT instead uses LMMs that reason over visual information with scene graphs as a bridge between visual and language domains.
  • Multimodal Prompting Methods: VidIL reasons over video-frame captions, DDCoT reasons over image captions, and Multimodal-CoT requires fine-tuning on ground-truth natural-language reasoning data.
  • CCoT Positioning: CCoT differs by using generated scene graphs rather than captions as its Chain-of-Thought reasoning step.The method is zero-shot, operates at inference time, and is applicable across a wide range of LMM architectures.
  • Compositionality: Strong LMMs still struggle with compositional visual understanding, including identifying object attributes and inter-object relations.The paper links this difficulty to less-compositional bag-of-objects representations and evaluates whether structured prompting improves performance.

3. Compositional Chain-of-Thought

CCoT turns an LMM’s visual and textual inputs into a two-stage process: generate a task-relevant JSON scene graph, then use it with the image and prompt to generate the answer. The second stage adds context and output-format instructions so the method can support different response types.

  • Preliminaries: A standard LMM encodes an image and text task prompt into a shared space before producing a text response.The image uses a trainable vision encoder, while the task prompt uses a fixed language embedding.
  • Overview: CCoT extends standard LMM prompting with scene-graph generation as an intermediate multimodal reasoning step.The method is purely prompting-based, requiring neither fine-tuning nor annotated scene graphs.
  • Step 1: Scene Graph Generation: The first CCoT step generates a scene graph from the image and task prompt, organizing objects, attributes, and relationships.Conditioning on the task prompt keeps the graph relevant because scene graphs can contain long-tailed, task-irrelevant information.
  • Step 1: Scene Graph Generation: The scene-graph prompt is formatted as JSON to standardize visual information and facilitate interpretation by the LMM.
  • Step 2: Response Generation: The second step prompts the LMM with the image, task prompt, generated scene graph, a context instruction, and an answer-extraction instruction.The extraction instruction can enforce letters for multiple-choice tasks or be modified or removed for short answers and descriptions.

4. Experiments and Results

The experiments evaluate CCoT across four LMMs and compositional, general multimodal, and ablation benchmarks. CCoT improves performance broadly, while structured SG prompting and an optimal SG length are important design choices.

  • Experimental Setup: CCoT is evaluated on InstructBLIP-13B, LLaVA-1.5-13B, Sphinx, and GPT-4V across multimodal reasoning and visual-linguistic compositional tasks.The study compares these models with prompting baselines on SEEDBench, MMBench, Winoground, WHOOPS!, and LLaVA-Bench In-the-Wild.
  • Experimental Setup: The main results table covers SEEDBench, MMBench, Winoground, and WHOOPS!, with supervised scene-graph-finetuned models marked separately.Winoground reports text, image, and group scores; SEEDBench-Image is abbreviated as SEED-I.
  • Main Results: CCoT outperforms base models across several benchmarks and improves all four tested LMMs on Winoground and WHOOPS!.GPT-4V-CCoT significantly improves over the previous Winoground SOTA SGVL without instruction tuning, while zero-shot language CoT often degrades compositional performance.
  • Main Results: CCoT also improves SEEDBench image splits, MMBench, and LLaVA-Bench In-the-Wild, with gains of at least 1% reported across the evaluated comparisons.LLaVA-Bench evaluates open-ended long-form visual question answering, creating a greater language-response burden than the other benchmarks.
  • Ablations: Requiring object bounding boxes decreases SEEDBench-Image weighted-average accuracy by 1.6%, indicating that exact object locations are not beneficial in this setting.This is an ablation of the generated scene-graph prompt.
  • Ablations: Removing JSON formatting lowers performance by 2.0%, showing that a common systematic structure helps LMMs use generated scene graphs.The ablation compares CCoT with and without an explicit JSON-format requirement.
  • Ablations: 256 tokens is the optimal tested scene-graph length; using 1024, 512, or 128 tokens lowers performance by 0.6%, 0.5%, or 0.3%, respectively.The results support compact textual SGs while indicating that a minimum amount of information is needed for answering.

5. Conclusion

The paper concludes that CCoT extracts compositional information from LMMs through zero-shot generated scene graphs, improving compositional and general multimodal reasoning without fine-tuning or annotated SG data. Ablations further support structured SGs, JSON formatting, and an appropriate SG length.

  • Conclusion: CCoT enhances compositional visual and multimodal reasoning without fine-tuning or ground-truth annotated scene-graph data.The method generates scene graphs zero-shot and uses them within a reasoning prompt.
  • Conclusion: Improvements extend across Winoground, WHOOPS!, SEEDBench, MMBench, and LLaVA-Bench In-the-Wild.The conclusion presents these results as evidence of effectiveness across diverse task types.
  • Conclusion: Ablations identify structured SGs over captions, JSON formatting, and optimal SG length as important for visual compositional and multimodal reasoning.These design choices are linked to the method’s observed performance improvements.

6. Limitations

CCoT improves multimodal and compositional visual reasoning across models and benchmarks, but its applicability is constrained by context length and task type.

  • CCoT is limited by the current context lengths of the LLMs used by LMMs.
  • Scene graphs are less useful for multimodal tasks emphasizing language over visual reasoning, such as document understanding.
  • The authors recommend caution when applying this machine learning method.

Supplementary Material

The supplementary material provides additional experiments, implementation details, datasets, and qualitative visualizations of CCoT.

  • The supplementary material adds experiment results, implementation details, datasets, and qualitative visualizations.Section A covers experiments, Section B covers implementation details, and Section C provides qualitative visualizations.

A. Additional Experiment Results

Additional experiments further examine the benefits of CCoT, including results on MMBench Perception Splits.

  • A.1. Additional Ablations: Additional ablations further demonstrate the benefits of CCoT.
  • A.2. Additional Results: CCoT is also evaluated on MMBench Perception Splits.

A.1. Additional Ablations

The ablations show that CCoT benefits from combining image information with scene graphs and from including objects, attributes, relationships, and the scene-graph terminology in its prompt.

  • Image and Scene Graph Inputs: 26.1% performance decrease after masking image tokens shows that CCoT benefits from combining the image with the scene graph.
  • Impact of Individual Compositional Characteristics: Removing objects, attributes, or relationships reduces accuracy gains, while combining all three achieves the best result.On SEEDBench, the reductions are 1.5/0.8/1.2%; on Winoground Text, they are 3.5/3.5/2.25%, respectively.
  • Prompt Terminology: -2.1% accuracy after replacing “scene graph” with “description” suggests pretrained scene-graph knowledge helps structured compositional reasoning.
  • COCO and Visual Genome Data Overfitting: CCoT shows no substantial difference from zero-shot performance on GQA and VL Checklist, where COCO or Visual Genome images are heavily reused.The reported differences are -.8% on GQA and -.6% on VL Checklist.
  • SEEDBench-Image Ablations: Table 4 reports split-level ablation results across SEEDBench-Image image categories and includes a weighted average.
  • Detailed SEEDBench Results: Table 5 reports detailed split-level results across SEEDBench image categories, including counting, attributes, location, spatial relation, and visual reasoning.

A.2. Additional Results

Additional evaluations show that CCoT provides small gains on the open-ended LLaVA-Bench, while detailed split-level results are reported for SEEDBench-Image and MMBench.

  • Detailed Split Results: Detailed results are provided for individual SEEDBench-Image splits and MMBench Perception and Reasoning splits.These split-level results appear in Tables 5, 6, and 7.
  • LLaVA-Bench Results: LLaVA-Bench improves by +0.4% for InstructBLIP and +0.3% for LLaVA-1.5 with CCoT.The benchmark evaluates long-form conversational responses on a small, hand-designed set of images and questions.
  • LLaVA-Bench Results: CCoT avoids the roughly 10% decrease observed with ZS-CoT on LLaVA-Bench.The contrast suggests different behavior on open-ended visual chat evaluation.

B. Additional Implementation Details

The appendix describes evaluation datasets, their task structures, and the inference procedures used to assess CCoT across multimodal benchmarks.

  • Inference Details: Experiments use eight NVIDIA RTX 6000 GPUs for larger benchmarks, while Winoground and MM-Vet run on a single GPU.Default model generation parameters are used apart from output token generation length.
  • SEED-Bench: SEED-Bench contains 19k human-annotated multiple-choice questions spanning 12 evaluation dimensions.Only its image splits are used for evaluation, with the official dataset and the method from the main paper.
  • MMBench: MMBench combines a curated dataset covering varied abilities with CircularEval and ChatGPT integration for systematic assessment.The appendix characterizes it as improving benchmark scale and assessment depth.
  • Winoground: Winoground contains 400 image-caption sets designed to test compositional matching beyond superficial correlations.It reports text, image, and group scores for matching images and captions with similar compositional structure.
  • WHOOPS!: WHOOPS! uses 500 synthetic commonsense-violating images with 10874 annotations and evaluates explanation, captioning, cross-modal matching, and VQA.Its scenarios target temporal, biological, cultural, physical, and social reasoning.
  • LLaVA-Bench: LLaVA-Bench evaluates detailed generalized chat responses by comparing model outputs with GPT-4-generated responses.The benchmark uses multimodal questions over varied images.

C. Qualitative Visualizations

The appendix presents qualitative success and failure cases for LLaVA-1.5-CCoT, alongside additional examples across several multimodal benchmarks.

  • Success and Failure Cases: The appendix includes further qualitative success and failure cases for LLaVA-1.5-CCoT.
  • Data and Impact: The datasets used are reported to contain no harmful or offensive content, though caution is recommended for the machine learning method.
  • Additional Examples: Figure 4 presents additional CCoT outputs on MMBench, WHOOPS!, and LLaVA-Bench.
Loading 2311.17076v3…