Source-linked AI summary
CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, Jiebo Luo
TL;DR
LMMs struggle to preserve fine-grained details and relationships across multiple images. CoCoT has models compare image similarities and differences before answering detailed questions, and the paper reports improvements for open- and closed-source models. The approach may introduce unnecessary noise because it requires other images for comparison.
Problem
LMMs can lose fine-grained visual detail and struggle to discern relationships across multiple images, which require joint consideration of the images.
Source: §1 · p. 1Method
CoCoT prompts LMMs to compare similarities and differences among image inputs, then answer detailed multi-image questions using those comparisons.
Source: §1 · p. 1, §3.2 · p. 3Results
CoCoT significantly improves performance for both open-source and closed-source models and improves image-to-image matching and multi-image-to-text performance.
Source: §1 · p. 2, §5 · pp. 5–6Takeaways & Limitations
The paper presents CoCoT as a prompting strategy for enhancing LMMs’ ability to discern fine-grained details in multi-image tasks.
Source: §5 · p. 5Because CoCoT requires other images for comparison, it might introduce unnecessary noise.
Source: §5 · p. 6
Abstract
from arXiv · showhide
When exploring the development of Artificial General Intelligence (AGI), a critical task for these models involves interpreting and processing information from multiple image inputs. However, Large Multimodal Models (LMMs) encounter two issues in such scenarios: (1) a lack of fine-grained perception, and (2) a tendency to blend information across multiple images. We first extensively investigate the capability of LMMs to perceive fine-grained visual details when dealing with multiple input images. The research focuses on two aspects: first, image-to-image matching (to evaluate whether LMMs can effectively reason and pair relevant images), and second, multi-image-to-text matching (to assess whether LMMs can accurately capture and summarize detailed image information). We conduct evaluations on a range of both open-source and closed-source large models, including GPT-4V, Gemini, OpenFlamingo, and MMICL. To enhance model performance, we further develop a Contrastive Chain-of-Thought (CoCoT) prompting approach based on multi-input multimodal models. This method requires LMMs to compare the similarities and differences among multiple image inputs, and then guide the models to answer detailed questions about multi-image inputs based on the identified similarities and differences. Our experimental results showcase CoCoT's proficiency in enhancing the multi-image comprehension capabilities of large multimodal models.
1. INTRODUCTION
LMMs struggle to preserve fine-grained details and relationships across multiple images, while existing prompting methods are less effective for multi-image than single-image understanding. The paper introduces CoCoT and reports improvements for open- and closed-source models.
1. INTRODUCTION
- LMMs face both loss of visual detail and difficulty reasoning about relationships among multiple images, where descriptions must capture individual elements and their spatial and contextual ties.The introduction notes that existing prompting strategies show proficiency in single-image contexts but struggle with relationships across multiple images.Source: §1 · p. 1
- CoCoT asks LMMs to identify similarities and differences among image inputs, then use those comparisons to answer detailed questions about the images.The paper evaluates it on image-to-image matching and multi-image-to-text matching.Source: §1 · pp. 1–2
- Figure 1 compares multimodal prompting strategies by highlighting unique components in their responses, using GPT-4V as the example model.Source: Figure 1 · p. 2
- CoCoT produces significant improvements for both open-source and closed-source models, while most current models perform poorly on fine-grained multi-image tasks.The introduction identifies image-to-image matching and multi-image-to-text matching as evaluation tasks.Source: §1 · pp. 1–2
2. RELATED WORK
The related work contrasts LMM architectures with differing multi-image capabilities and reviews prompting methods for multimodal reasoning. The examples illustrate responses and prompts involving image descriptions and cross-image comparisons.
2. RELATED WORK
- Some open-source LMMs are architecturally limited to single-image reasoning, while OpenFlamingo and MMICL process multiple image features; GPT-4V and Gemini are described as capturing broader scene context.Fine-tuning is a common performance-enhancement technique, but applying it to LMMs presents computational challenges.Source: §2 · p. 2
- Multimodal prompting methods include zero-shot, few-shot, and Chain-of-Thought approaches, but current prompts can miss intricate relationships between visual and language information in multi-image inputs.The passage gives the example of missing the boy throwing the ball and proposes holistic consideration of all input images as a direction for improvement.Source: §2 · p. 2
- The examples contrast object-and-scene descriptions with prompts comparing image similarities and differences, including car-and-milk scenes and car-damage cases.The figure caption distinguishes CCoT image-information generation from CoCoT comparisons across images with large or small domain gaps.Source: §2 · p. 3
3. CONTRASTIVE CHAIN-OF-THOUGHT
Existing CoT prompting can miss fine details in image-rich questions: text-based decomposition does not directly acquire image details, while image-based scene graphs may focus on basic objects. CoCoT compares images first, then answers using the comparison.
3.1. Motivation Analysis
- Text-based DDCoT does not directly acquire detailed image information, while image-based CCoT can extract basic objects yet overlook question-relevant details.The examples include missing that a child in the second image is not throwing a basketball and producing scene graphs unrelated to the question.Source: §3.1 · p. 3
3.2. Methodology
- CoCoT first prompts LMMs to compare similarities and differences across images, then directs them to answer questions using insights from that comparison.The motivation describes comparisons as helping highlight contrasts such as damage severity and possible causes, including when images have different domains.Source: §3.2 · p. 3
4. EXPERIMENTS AND RESULTS
The experiments evaluate CoCoT and other prompting strategies on fine-grained image matching tasks. CoCoT improves results across both tasks, while the ablation indicates that combining similarity and difference prompts gives the best results for the tested models.
4.1. Experiment Setup
- Experiments use Raven-50 and Factify2 for image-to-image matching, Winoground for multi-image-to-text matching, and compare CoCoT with DDCoT, CCoT, and standard prompting in a zero-shot setting.The evaluated models include OPENFLAMINGO, MMICL, GPT-4V, and GEMINI; GPT-4V is compared only under standard and CoCoT prompting because of API restrictions.Source: §4.1 · p. 4, §4.2.1 · p. 4
4.2.1. Image-to-image Matching; 4.2.2. Multi-image-to-text Matching
- CoCoT improves both image-to-image and multi-image-to-text matching, reaching state-of-the-art performance in most image-to-image cases and outperforming other CoT methods in most text-matching scenarios.The supplied tables identify the accuracy results for the two tasks; the passages do not provide their cell values.Source: Table 1 · p. 4, §4.2.1 · pp. 4–5, Table 2 · p. 5
4.2.1. Image-to-image Matching
- On Raven-50, GPT-4V performs best and GEMINI worst; on Factify-V, prompted GEMINI approaches GPT-4V.Factify-V uses natural images, while Raven-50 comprises non-natural images made up of shapes.Source: §4.2.1 · p. 4
4.2.2. Multi-image-to-text Matching
- CoCoT substantially improves multi-image-to-text matching and beats other CoT methods in most scenarios, although all current models remain significantly below human performance.The authors attribute mediocre performance across tasks in part to differences between visual-encoder and language-model latent spaces and to LLM generalization issues.Source: §4.2.2 · p. 5
4.3. Ablation Study
- For GEMINI, either similarity-only or difference-only prompts help, but less than the full prompt; for MMICL, differences alone slightly lower performance, while both prompts yield its best results.The ablation compares similarity and difference prompt variants.Source: §4.3 · p. 5, Table 3 · p. 5
5. DISCUSSIONS AND CONCLUSIONS
The paper presents CoCoT as a prompting strategy for detailed multi-image tasks and reports improved performance on both matching tasks. It notes that comparison images may introduce noise and identifies more complex scenarios as an area for future refinement.
5. DISCUSSIONS AND CONCLUSIONS
- CoCoT improves performance in image-to-image matching and multi-image-to-text tasks.The conclusion describes it as a prompting approach for discerning fine-grained details across multiple images.Source: §5 · pp. 5–6
- Because CoCoT requires other images for comparison, it might introduce unnecessary noise; the authors call for refinement in more complex scenarios.They also suggest integrating it with other AI technologies in future work.Source: §5 · p. 6
A.1. Winoground
Winoground tests fine-grained image-text matching using groups of similar image-caption pairs. Its captions reuse the same words in different sequences, requiring models to distinguish subtle differences and implied references.
A.1. Winoground
- Winoground contains 400 groups, each with two similar image-caption pairs.The task tests whether LMMs can comprehend fine-grained image information in relation to text.Source: §A.1 · p. 8
- Captions use the same words in different sequences, requiring models to detect subtle differences and understand implied references.Source: §A.1 · p. 8
- The dataset tests both matching a caption to one of two images and matching one image to one of two text descriptions.Source: §A.1 · p. 8
A.2. Raven-50
Raven-50 tests whether LMMs can use visual acuity and logical reasoning to identify relationships among images and select the correct answer image.
A.2. Raven-50
- Raven-50 presents either 3 or 8 input images and 6 candidate answer images, with one correct solution to identify.The test assesses nonverbal reasoning through connections among images.Source: §A.2 · p. 8
- OPENFLAMINGO and MMICL are evaluated by logits for each image pair, while GPT-4V and GEMINI choose an answer directly and are scored for accuracy.Source: §A.2 · p. 8
A.3. Factify-V
Factify-V is a sampled image-only subset of Factify2 used to test whether models can judge contextual entailment between image pairs.
A.3. Factify-V
- Factify-V samples 500 Factify2 test cases, with 100 cases from each of the dataset’s five categories.Source: §A.3 · p. 8
- The original Factify2 pairs contain a claim and document, each combining an image, text, and OCR-generated text, and use five labels.Source: §A.3 · p. 8
- For Factify-V, experiments use only images, regroup labels as “support image” and “refute,” and ask whether each image pair is contextually entailed.Source: §A.3 · p. 8
B. ADDITIONAL DISCUSSIONS OF MOTIVATION
The discussion contrasts prior CoT approaches with CoCoT’s focus on comparing multiple images, especially when they are similar.
B. ADDITIONAL DISCUSSIONS OF MOTIVATION
- DDCoT decomposes the question and image information into sub-questions, obtains sub-answers, then uses them to answer the original question.Source: B. ADDITIONAL DISCUSSIONS OF MOTIVATION · p. 8
- CCoT first generates a Scene Graph from image information, then combines that information with the user’s question to produce an answer.Source: B. ADDITIONAL DISCUSSIONS OF MOTIVATION · p. 8
- The authors say these approaches do not effectively capture detail, motivating CoCoT to identify similarities and nuanced differences between images.CoCoT is intended to support more in-depth and accurate interpretation of visual content.Source: B. ADDITIONAL DISCUSSIONS OF MOTIVATION · p. 8
C. MORE EXAMPLES
The section provides sampled dataset questions and example GPT-4V responses using CoCoT for Raven-50 and Factify-V.
C. MORE EXAMPLES
- Fig. 5 and Fig. 6 show example GPT-4V responses generated via CoCoT on Raven-50 and Factify-V, respectively.Source: C. MORE EXAMPLES · p. 8, Figure 5 · p. 10, Figure 6 · p. 11
- Fig. 4 presents sampled questions from the Raven-50, Factify-V, and Winoground datasets.Source: Figure 4 · p. 9