CaMML: Context-Aware Multimodal Learner for Large Models

1 Introduction

• LMM inference is constrained by knowledge in model parameters, while inflexible designs make generalization from contextual examples challenging.Source: §1 · p. 1
• Integrating retrieved context is difficult because the number of examples and their interleaved modalities can vary, producing lengthy, heterogeneous inputs.Source: §1 · p. 2
• CaMML uses a hierarchical design: cross-attention connects text and images within each example, then combines contextual information with learnable fixed-length tokens for LLM input.Source: §1 · p. 2
• CaMML achieves state-of-the-art performance across diverse multimodal benchmarks, outperforming LLaVA-1.5 and many other large multimodal models.Source: Figure 1 · p. 1, §1 · p. 2
• The introduction frames CaMML as a lightweight approach for processing extremely long multimodal contexts and reports analyses and case studies of challenging real-world examples.Source: §1 · p. 2

Large Multimodal Models

• Multimodal models commonly add vision encoders to language models, while other approaches map modalities into a shared input-output space and train from scratch.Source: §2 · p. 2

Multimodal Few-shot Learning

• Multimodal few-shot learning includes Flamingo and Frozen, while retrieval-augmented generation uses external repositories to supply up-to-date, domain-specific expertise.Source: §2 · p. 2
• MuRAG concatenates retrieved image and text embeddings for multimodal question answering, ReViLM retrieves text for captioning, and RA-CM3 augments text-image generation.Source: §2 · p. 3
• CaMML differs by targeting lightweight, efficient processing of multimodal context, with the stated objective of improving generalization and reasoning from in-context examples.Source: §2 · p. 3

3.1.1 Datastore and Context Retriever

• ImageBind embeddings index datastore samples in Faiss; the frozen retriever selects the top N contexts for a multimodal query, with query-text/context-image similarity retrieving paired images and texts.Source: §3.1.1 · p. 3, §3.1.2 · p. 3

3.1.2 Multimodal CaMML Perceiver

• Its hierarchical Vision, Language, and Context Perceivers process image and text together for each sample, then condense long outputs into M tokens (M < L) while accommodating dynamic context counts.The Vision and Language Perceivers use cross-attention between modalities; the Context Perceiver compresses their outputs.Source: §3.1.2 · pp. 3–4

3.2 Model Training

• Using Vicuna-7B and Vicuna-13B backbones, training freezes the retriever and vision encoder and minimizes causal language-modeling loss; Table 1 covers ScienceQA train/test accuracy.CaMML is fine-tuned on the train split and evaluated on the test split.Source: §3.2 · pp. 4–5, Table 1 · p. 5

4 Experiment

• Across Tables 1–3, CaMML-7B and CaMML-13B establish state-of-the-art results across multimodal benchmarks.The experiments describe the settings in subsequent sections and supplementary material.Source: §4 · p. 5

4.1 Multimodal Reasoning on ScienceQA

• On ScienceQA, CaMML achieves the best average performance, while CaMML-13B performs strongly on image questions against LLaVA-13B and MMCoTLarge.The authors report a noticeable margin over the baselines overall and a wide margin on IMG questions.Source: §4.1 · p. 5

4.2 Multimodal Instruction Tuning

• Instruction tuning uses LLaVA-665K for both training and the datastore, with three relevant multimodal samples retrieved at inference.Training varies each sample's context count from one to three (“mixed-shots training”).Source: §4.2 · p. 5
• CaMML achieves the best performance on 10/11 tasks among 11 listed benchmarks; CaMML-13B performs best across all 11, surpassing LLaVA-1.5, which uses the same training data.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Multimodal Few-shot Learning

• With three shots, CaMML-13B achieves the best performance on 5/5 captioning and VQA tasks, outperforming Flamingo-80B evaluated with 32 shots.The comparison covers captioning and visual question answering.Source: Table 3 · p. 6, §4.3 · p. 7

5.1.1 Contribution of Each Components

• Removing perceivers markedly reduces ScienceQA performance, while shared and unshared Context Perceiver weights perform nearly identically; shared weights save computation.The component ablations are reported for CaMML-7B on ScienceQA-test.Source: §5.1.1 · p. 7

5.1.2 Impact of Hyperparameters

• Two and twelve perceiver layers yield similar performance; larger hidden sizes are more favorable, while increasing query number M does not consistently improve results.The authors select two layers because the model is smaller.Source: §5.1.2 · p. 8
• Increasing inference context count N does not consistently improve performance, while mixed-shots training shows potential for consistently better results than fixed-N training.CaMML accommodates many shots; the authors suggest longer contexts may add complexity and convolute inference.Source: §5.1.2 · p. 8

5.1.3 Inference Cost

• As context samples grow, CaMML's efficiency and memory benefits over directly feeding all context tokens to the LLM become more significant.The analysis compares inference speed and memory footprint.Source: §5.1.3 · p. 8

5.2.1 The Importance of Context-Awareness

• Context examples support accurate answers through relevant examples and analogous domain-specific samples, including multimodal analogy-based learning.The comparison uses a Great Wall sketch and a butterfly metamorphosis depiction.Source: §5.2.1 · p. 8, Figure 5 · p. 8

5.2.2 Handling Image Sequences

• Without task-specific training, CaMML answers image-sequence questions accurately using the last image as the query and preceding images as context.The reported comparisons find LLaVA-1.5 and GPT-4V struggle with the sequences and lack access to real-time updates.Source: §5.2.2 · p. 8, Figure 6 · p. 9

5.2.3 Tackling Multimodal Hallucination

• For muffin–Chihuahua and fried-chicken–labradoodle illusions, CaMML answers accurately using relevant context, while LLaVA-1.5 and GPT-4V hallucinate.The image grid is treated as a single image.Source: §5.2.3 · p. 9, Figure 7 · p. 9

• CaMML-13B achieves state-of-the-art results across more than ten prominent multimodal benchmarks, surpassing previous methods by a substantial margin.Source: §6 · p. 9
• CaMML-7B and CaMML-13B draw on analogous, domain-specific, up-to-date context samples to make grounded inferences.Source: §6 · p. 9
• Lightweight multimodal perceivers integrate context samples and support efficient processing of lengthy context tokens.Source: §6 · p. 9

• Irrelevant retrieved examples may impede CaMML’s performance.Source: §7 · p. 9
• The datastore may not cover all domains, including medical images, and substantial distribution mismatch can limit retrieval effectiveness.Source: §7 · p. 9
• CaMML requires substantial training on large language models and access to a sizable datastore to demonstrate its effectiveness.Source: §7 · p. 9

8 Potential Risks

• Environmental unfriendliness and increased computational costs are potential risks, particularly with larger models.Source: §8 · p. 9
• Although users cannot directly access the encapsulated datastore, the I/O API could be used to instruct the LLM to generate concerning or privacy-leaking text.Source: §8 · p. 10

A.1 Experimental Setup

• The experiments use LLaVA-665K for instruction tuning, ScienceQA’s train split for finetuning and ablation, and a set of multimodal benchmarks for evaluation.Evaluation also includes captioning, visual question answering, and grounding datasets.Source: §A.1 · p. 13
• Evaluation uses accuracy for several benchmarks, GPT4-assisted scores for MM-Vet, F1 for POPE, and summed perception-subtask scores for MME.Source: §A.1 · p. 13
• ScienceQA finetuning runs for 12 epochs, while instruction finetuning runs for 1 epoch; Vicuna, CLIP-ViT-L-14, and ImageBind are used in the model and context setup.Source: §A.1 · pp. 13–14

A.2 CaMML Retrieval Setup

• The retrieval setup embeds datastore images with ImageBind, indexes them with Faiss, and selects top-k contexts by query similarity.Quantitative experiments compare text-query embeddings with a vision datastore, while qualitative analyses use vision-query embeddings against that datastore.Source: §A.2 · p. 14

A.3.1 Multimodal Task Performance

• 82.0 accuracy on A-OKVQA is reported for CaMML.The passage describes this as exceptional performance on the task.Source: §A.3.1 · p. 14
• CaMML also shows good ability on visual grounding tasks such as RefCOCO/+/g.Source: §A.3.1 · p. 14

A.3.2 Finegrained Evaluation

• CaMML-13B achieves 8 out of 12 state-of-the-art results among large multimodal models on fine-grained evaluations.The evaluations cover MMBench and MMBench-CN.Source: §A.3.2 · p. 14

Image Generation

• CaMML generates content from multimodal contextual sources in image-generation examples.Figures 8 and 9 show city-view and book-cover examples created via the DALL·E3 API.Source: B Additional Cases · p. 14, Figure 8 · p. 15, Figure 9 · p. 15
• With context, generated images are likely to mix traditional Chinese-style buildings with cyberpunk style; without context, they are more likely to show modern skyscrapers only.Source: B Additional Cases · p. 14, Figure 8 · p. 15, Figure 9 · p. 15

Implicit-Description Localization

• CaMML identifies objects from implicit descriptions, including an airplane described as carrying hundreds of souls skyward and a mirror described through duplicated dogs.Figure 10 illustrates localization from such descriptions.Source: B Additional Cases · p. 14, Figure 10 · p. 16