PaperlayerView exact revision on arXiv

Source-linked AI summary

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, Baobao Chang

arXiv:2309.07915v3cs.CLcs.AIcs.CV

TL;DR

Most VLMs struggle with complex multi-modal prompts containing multiple images, including text-to-image references and relationships among images. MMICL addresses this with a context scheme, multimodal in-context dataset, and two-stage training, achieving state-of-the-art benchmark performance while improving multimodal ICL and mitigating language bias.

  • Problem

    Most VLMs struggle to understand complex multi-modal prompts with multiple interleaved images and text, limiting their effectiveness on downstream vision-language tasks.

  • Method

    MMICL combines a context scheme with image declarations and proxy tokens, a MIC dataset, and multimodal in-context tuning to model image references and inter-image relationships.

  • Results

    MMICL achieves state-of-the-art performance across vision-language benchmarks, with 13-point Winoground and 12-point RAVEN improvements.

  • Takeaways & Limitations

    MMICL improves complex multi-modal prompt understanding and multimodal in-context learning while mitigating language bias associated with hallucinations.

Abstract

from arXiv · show

Since the resurgence of deep learning, vision-language models (VLMs) enhanced by large language models (LLMs) have grown exponentially in popularity. However, while LLMs can utilize extensive background knowledge and task information with in-context learning, most VLMs still struggle with understanding complex multi-modal prompts with multiple images, making VLMs less effective in downstream vision-language tasks. In this paper, we address the limitation above by 1) introducing vision-language Model with Multi-Modal In-Context Learning(MMICL), a new approach to allow the VLM to deal with multi-modal inputs efficiently; 2) proposing a novel context scheme to augment the in-context learning ability of the VLM; 3) constructing the Multi-modal In-Context Learning (MIC) dataset, designed to enhance the VLM's ability to understand complex multi-modal prompts. Our experiments confirm that MMICL achieves new state-of-the-art zero-shot performance on a wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Our analysis demonstrates that MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges the impressive ICL ability. Furthermore, we observe that MMICL successfully alleviates language bias in VLMs, a common issue for VLMs that often leads to hallucination when faced with extensive textual context. Our code, dataset, dataset tool, and model are available at https://github.com/PKUnlp-icler/MIC

1 INTRODUCTION

MMICL addresses VLMs’ difficulty with complex prompts containing multiple interleaved images and text, including references and relationships across images. It introduces a context scheme and MIC dataset to improve multimodal in-context learning, achieving strong benchmark results and reducing language bias.

  • Research Gap: Most VLMs struggle with complex multi-modal prompts containing multiple images, unlike LLMs that benefit from in-context learning.Prior models mainly target single-image queries, while existing multi-image pretraining data lacks sophisticated interleaved image-text prompts.
  • Research Gap: Existing VLMs have difficulty resolving text-to-image references when different words refer to different images.Web-crawled training data may lack explicit references linking text to particular images.
  • Research Gap: VLMs also struggle to infer spatial, temporal, and logical relationships among multiple images.Internet-collected pretraining data often lacks close connections among images, limiting multi-image reasoning.
  • Contributions: MMICL introduces a multimodal approach, context scheme, and dataset designed to support multi-image inputs, image references, and in-context learning.The scheme adds image declarations and proxy tokens, while MIC adapts data from existing datasets for training.
  • Results: 13 points on Winoground and 12 points on RAVEN quantify MMICL’s gains in text-to-image reference and multi-image reasoning, respectively.The paper also reports strong multimodal in-context learning across tasks and reduced language bias associated with hallucinations.

2 MMICL

MMICL treats image and text representations equally, uses image declarations and proxy tokens to organize interleaved inputs, and trains on unified multimodal in-context data. Its MIC dataset and two-stage training target reference understanding, inter-image reasoning, and multimodal ICL.

  • Model Architecture: MMICL treats image and text representations equally, allowing multiple images and text in arbitrary order and quantity.Image declarations establish references between image embeddings and textual descriptions.
  • Context Scheme: Image declarations assign each image a unique proxy token and use natural-language prompts to link text with the appropriate image.The declarations remain textual, preserving flexibility in prompt placement.
  • Context Scheme: Interconnected multi-image data encode spatial, logical, and temporal relationships to improve understanding of multi-image queries.The construction derives video frames and other related-image examples for the context scheme.
  • Context Scheme: Few-shot exemplars sampled from task data are combined with inputs to convert different tasks into a unified multimodal in-context format.Diverse instructions and templates expand the variety of multimodal demonstrations.
  • MIC Dataset: MIC combines image declarations, closely related multi-image data, and task-diverse in-context data from 16 training datasets across 8 categories.The test set contains 18 datasets across 10 categories, and construction uses existing datasets plus instruction refinement.
  • Training Paradigm: MMICL uses two training stages: frozen-encoder alignment followed by multimodal in-context tuning on MIC data.Stage I trains the visual prompt generator and projection layer; Stage II targets image-text references, inter-image relationships, and multimodal ICL.

3 EXPERIMENT

MMICL is evaluated on general, complex reasoning, few-shot, hallucination, and ablation benchmarks. Results show strong performance on MME, MMBench, Winoground, RAVEN, in-context learning, and multi-image tasks.

  • General Performance: MMICL achieves the best average scores among compared VLMs on MME cognition and perception tasks and strong performance on MMBench.MME covers 14 cognition and perception subtasks, while MMBench evaluates diverse VLM skills.
  • In-Context Learning: MMICL outperforms competing VLMs on held-in and held-out few-shot tasks, including VizWiz gains of 15.38 and 14.98 points over Flamingo-9B and KOSMOS-1.VizWiz was not included in training data, supporting evaluation of generalization from exemplars.
  • Hallucination and Language Bias: MMICL performs equally well on ScienceQA-IMG questions requiring images and questions not requiring images, indicating reduced language bias.The study also reports impressive object-hallucination performance in Appendix L.
  • Training Paradigm: Multi-modal in-context tuning improves performance across model types and sizes, especially on tasks involving multiple images.The ablation attributes gains to Stage II and reports strong performance across video datasets.
  • Context Scheme: The context scheme's components collectively drive MMICL's superiority, and removing any component does not guarantee superior performance.The ablation reports that each component contributes to different model capabilities.

4 CONCLUSION

The paper concludes that current VLMs remain limited on complex multi-modal prompts with multiple images. MMICL addresses these limitations through a multi-modal architecture, context scheme, and MIC dataset, achieving strong general and complex-reasoning performance.

  • Conclusion: MMICL is introduced to handle multi-modal inputs, relationships among multiple images, and text-to-image references.The paper extends this approach to multi-modal in-context learning.
  • Conclusion: The paper reports new state-of-the-art performance on general VLM benchmarks and complex multi-modal reasoning benchmarks.The benchmark summary includes MME, MMBench, Winoground, and RAVEN-related evaluations.
  • Limitations of Existing VLMs: Current VLMs often focus on single-image prompts, while multi-image systems can still perform poorly or struggle to reference specific images.Flamingo supports few-shot learning and multi-image inputs but remains unable to explicitly point to images without a cross-attention mask.
  • Data Resources: The MIC training resources span multiple task categories and datasets covering image, video, question-answering, captioning, classification, dialog, and reasoning.The training data include image-text interleaved and in-context learning data, with datasets such as MS COCO, OK-VQA, VQAv2, and MSRVTT.

D DATA CONSTRUCTION

The data construction process converts existing annotated datasets into multi-modal training examples with explicit image references, interleaved images, and in-context demonstrations. It uses automated generation supplemented by instruction review and produces 5.8M MIC samples.

  • Instruction and Reference Construction: Each instance receives an image declaration, while annotators inspect samples and provide task instructions for template creation.The declarations make text-to-image references explicit and the inspection supports high-quality templates.
  • Instruction and Reference Construction: ChatGPT rewrites task instructions into varied templates, which are then manually reviewed for quality.The process uses gpt-3.5-turbo according to the implementation note.
  • In-Context Data: Randomly selected templates are filled with original dataset inputs and combined with sequentially arranged demonstrations to form multi-modal contextual data.Demonstrations are sampled from the dataset and integrated with each input instance.
  • Interconnected and Video Data: The MIC dataset contains 5.8M vision-language question-answer samples built from video frames and VCR object crops.Eight frames are extracted per video, while VCR bounding boxes yield closely related cropped images.
  • Image Declaration: The image-declaration algorithm replaces each visual embedding with a reference containing its image index and proxy token while preserving interleaved text spans.The resulting representation explicitly associates each image with a corresponding declaration.
  • Automated Construction: Existing annotations enable automated construction for interconnected-image, in-context, and video data without additional manual annotation.VCR annotations provide images, bounding boxes, questions, and answers for the interconnected-image workflow.
  • MMICL Structure: MMICL encodes images with a vision encoder, extracts visual embeddings with a Q-former, projects them, and interleaves them with language representations.The architecture treats image and language representations equally in an interleaved image-text sequence.

F DATA BALANCE

The paper addresses training-data imbalance by sampling datasets according to the square root of their sizes. Its data-construction pipelines support interleaved video and in-context examples for MMICL.

  • Data Balance: Uniformly mixing datasets can overfit smaller datasets and underfit larger ones, so MMICL samples datasets with probabilities proportional to the square root of training-set size.The sampling probability is defined over D datasets with training counts N_1 through N_D.
  • Data Construction Pipelines: The in-context pipeline automatically constructs multi-model data with in-context examples from existing annotations and uses ChatGPT for instruction refinement.Instruction templates and source data are documented in the appendices.
  • Data Construction Pipelines: The video pipeline automatically constructs multi-model data from existing annotations, with ChatGPT used for instruction refinement.The cited figure describes automatic construction without human involvement.
  • MMICL Structure: The MMICL structure is illustrated as a dedicated model architecture for multi-modal in-context learning.The supplied figure identifies the illustration but provides no further architectural encoding.

H EXPERIMENT DETAILS

The experiments use established vision-language backbones and instruction-template tables to organize multimodal evaluation. Implementation details include frozen components, benchmark-specific templates, and a six-GPU training setup.

  • Implementation: MMICL uses FLAN-T5-XL and FLAN-T5-XXL as backbone language models, with vision and language components frozen during Stage I feature alignment.The reported data sources include COCO, CC3M, Visual Genome, CC12M, SBU, and LAION-400M, with captions generated by BLIP-large.
  • Implementation: In practice, MMICL uses BLIP-2 and InstructBLIP checkpoints as backbones, so Stage I is skipped.
  • Data formatting: Instruction templates cover general multimodal formatting and tasks including OKVQA, VQAv2, ST-VQA, WikiART, RefCOCO, MSRVTT, MSVD, GQV, VCR, and NLVR v2.The templates distinguish image embeddings from image-reference tokens for particular images.
  • Compute: Experiments run on six NVIDIA A40 GPUs with DeepSpeed zero2-offload; batch sizes are 10 for FLAN-T5-XL and 4 for FLAN-T5-XXL.The largest FLAN-T5-XXL model requires about two days for Stage II.

I MME BENCHMARK

MME evaluates perception and cognition across 14 subtasks, while the reported experiments compare MMICL with multiple VLM baselines and extend evaluation to video-language tasks. MMICL shows large gains on selected compositionality, multi-image reasoning, and video benchmarks.

  • MME benchmark: MME covers 14 subtasks spanning OCR, coarse- and fine-grained recognition, commonsense reasoning, calculation, translation, and code reasoning.
  • Benchmark comparisons: MMICL is compared with a broad set of VLM baselines on MME and MMBench, whose reported results are provided in benchmark tables.
  • Video-language evaluation: 10.86, 4.53, and 2.45 points improvement over the strongest baselines occurs on MSVD-QA, NExT-QA, and iVQA, respectively.The training dataset contained no videos, while the reported result concerns video-language tasks.

L OBJECT HALLUCINATION EVALUATION

The POPE evaluation tests MMICL alongside several VLMs for object hallucination performance.

  • POPE evaluation: The POPE benchmark compares MMICL with Shikra, InstructBLIP, MiniGPT-4, LLaVA, MM-GPT, and mPLUG-Owl.The corresponding result is reported in Table 24.

M.1 EVALUATION METRICS

The evaluation materials define VQA accuracy as the metric used across the reported metrics and provide table-based results for video-language and object-hallucination evaluations.

  • Video-language evaluation: Table 23 reports MMICL against other VLMs across video-language tasks, with BLIP-2 and InstructBLIP concatenating all frame embeddings at the prompt top.
  • Object hallucination evaluation: Table 24 reports the performance of different VLMs on the POPE benchmark.
  • Evaluation metrics: VQA accuracy is used across all metrics with the same VQA tools as the original VQA paper.

N.1 ABLATION STUDY ON CONTEXT SCHEME

The ablations show that MMICL’s context scheme derives its gains from the collective contribution of interrelated images, image declarations, and multi-modal in-context data. Removing components causes targeted losses in multi-image reasoning, image-text reference understanding, or task-specific perception and cognition.

  • Context Scheme: The context scheme’s superiority is driven by the collective impact of its design elements, rather than simply adding more pretraining data.Removing any component cannot guarantee superior performance, and each component contributes to different aspects of the model.
  • Interrelated Images: Removing interrelated images significantly decreases multi-image understanding on NLVR2 and Raven.Multi-modal in-context training preserves some multi-image ability, but does not match the full design.
  • In-context Format: Removing in-context format data prevents improvement on single-image tasks, while training with multiple images improves cognition but can reduce specific-object perception.The reported MME behavior reflects a trade-off between abstract reasoning and object recognition under this ablation.
  • Image Declaration: Removing image declarations sharply reduces performance on Winoground and other tasks requiring image-text or multi-image relationships.The ablation attributes the Winoground decline to lacking explicit modeling between image-text references.
  • In-context Learning Ability: The full context scheme improves in-context learning because all three designs are integrated, while gains vary by task format.ICL-only yields the largest VizWiz enhancements, whereas Flickr shows modest gains without multi-image instruction tuning.

S PERFORMANCE COMPARISON ON THE MULTI-IMAGE DATASETS

MMICL consistently outperforms other models on evaluated multi-image datasets and generalizes to an unseen Minecraft domain. Its image capacity remains bounded by the backbone context length, although reported video results extend beyond the tuning limit.

  • Performance Comparison: MMICL consistently outperforms other models on Raven-IQ and sampled NLVR2 and IconQA-img multi-image datasets.The comparison evaluates models with multi-modal instruction tuning using the same listed multi-image task families.
  • OOD Generalization: MMICL generalizes to an unseen Minecraft domain and tasks given a few examples.The paper attributes this result to diversified instruction templates intended to reduce overfitting to specific data structures.
  • Context Length Constraints: MMICL integrates up to eight images per instance during MIC tuning because the backbone context length restricts the number of input images.The model reportedly maintains strong video performance with inputs of up to 12/16 frames.
  • Model Scope: The approach’s effectiveness with decoder-only architectures has not been comprehensively investigated.The reported exploration primarily focuses on T5-series models.
Loading 2309.07915v3…