PaperlayerView exact revision on arXiv

Source-linked AI summary

Many-Shot In-Context Learning in Multimodal Foundation Models

Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, Andrew Y. Ng

arXiv:2405.09798v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Limited context windows had constrained investigation of how increasing the number of demonstrations affects multimodal in-context learning. The paper benchmarks multimodal models across 14 datasets and reports that many-shot ICL improves performance across most datasets, while gains differ across model families.

  • Problem

    Limited context windows constrained research on how increasing demonstrations affects multimodal ICL, particularly because images use many visual tokens.

    Source: §1 · p. 1

  • Method

    The authors benchmark API-access and open-weights multimodal models across 14 datasets and study many-shot ICL and query batching.

    Source: §1 · p. 2, §3 · p. 5, §3.1 · p. 5

  • Results

    Many-shot ICL brings substantial gains across most datasets, with less stable improvement for GPT-4o and no benefit from demonstrations for the tested open-weights models.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Many-shot ICL may let users adapt multimodal foundation models to new tasks and domains using demonstrations, including models that are difficult to fine-tune.

    Source: §5 · p. 9

    The study covers selected common multimodal tasks, context size limits use on datasets with several hundred or more classes, and private-model training data are undisclosed.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Large language models are effective at few-shot in-context learning (ICL). Recent advancements in multimodal foundation models have enabled unprecedentedly long context windows, presenting an opportunity to explore their capability to perform ICL with many more demonstrating examples. In this work, we evaluate the performance of multimodal foundation models scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro across 14 datasets spanning multiple domains (natural imagery, medical imagery, remote sensing, and molecular imagery) and tasks (image classification, visual QA, and object localization). We observe that many-shot ICL, including up to almost 2,000 demonstrating examples, leads to substantial improvements compared to few-shot (<100 examples) ICL across all of the datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly up to the maximum number of tested examples on many datasets. We also find open-weights multimodal foundation models like Llama 3.2-Vision do not benefit from the demonstrating examples, highlighting an important gap between open and closed multimodal foundation models. Given the high inference costs required for many-shot ICL, we also explore the impact of batching multiple queries in a single API call. We show that batching up to 50 queries can lead to performance improvements under zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on multiple datasets, while drastically reducing per-query cost and latency. Finally, while GPT-4o and Gemini 1.5 Pro achieve similar zero-shot performance across the datasets, Gemini 1.5 Pro learns more quickly than GPT-4o on most datasets. Our results suggest that many-shot ICL could enable users to efficiently adapt multimodal foundation models to new applications and domains. Our codebase is publicly available at https://github.com/stanfordmlgroup/ManyICL .

1 INTRODUCTION

Limited context windows had constrained tests of how additional demonstrations affect multimodal in-context learning; longer windows now make scaling these examples feasible. This section introduces a benchmark of many-shot ICL and reports gains, model differences, and query-batching findings.

1 INTRODUCTION

  • Close-weights multimodal models improve substantially with many-shot demonstrations; Gemini 1.5 Pro generally improves log-linearly, while GPT-4o’s gains are less stable.Figure 2 reports log-linear improvement for Gemini 1.5 Pro on 9 of 10 datasets.Source: §1 · p. 2, Figure 2 · p. 3
  • Open-weights models Llama 3.2-Vision and InternLM-XComposer2.5 do not benefit from demonstrations, unlike close-weights models.The authors identify this as a significant gap between open- and closed-weight multimodal foundation models.Source: §1 · p. 2
  • Gemini 1.5 Pro has higher ICL data efficiency than GPT-4o on most datasets as the number of demonstrations increases.Source: §1 · p. 2
  • Batching queries in one request can match or exceed single-query many-shot performance while reducing per-example latency and inference cost.Source: §1 · p. 2
  • In zero-shot settings, batching questions can substantially improve performance through domain calibration, class calibration, and autoregressively generated demonstrations.Source: §1 · p. 2
  • The authors release a tool for testing multimodal foundation models’ many-shot ICL capabilities.Source: §1 · p. 2

2 RELATED WORK

Prior work studied scaling in text-only ICL and benefits from few-shot multimodal ICL, but had not examined how expanded context windows affect many-shot multimodal ICL. Research on batching queries also established efficiency gains alongside possible performance degradation in longer contexts.

2 RELATED WORK

  • Text-only studies scaled demonstrations beyond 1,000 examples and found improvements across tasks, but did not compare models or study multimodal benchmarks.Source: §2 · p. 3
  • Prior multimodal ICL studies found benefits on out-of-domain tasks, but none used expanded context windows to test increasing demonstration counts.Source: §2 · p. 3
  • Batch-prompting studies report comparable or better performance with lower cost and latency, while one found degradation in longer contexts and proposed mitigations.Later variants group similar questions, combine tasks, or concatenate images into collages.Source: §2 · p. 3

3 METHODS

The study benchmarks multimodal models’ response to increasing in-context demonstrations across varied datasets and tasks, and examines query batching. It compares API-access and open-weights models, evaluates task-specific performance and ICL data efficiency, and includes batching ablations.

3.2 DATASETS

  • The benchmark covers 14 datasets across natural, medical, remote-sensing, and molecular imagery, with classification, visual QA, and object-localization tasks.Demonstration and test examples are randomly sampled without replacement; classification samples are stratified or balanced as applicable.Source: Table 1 · p. 4, §3.2 · p. 5

3 METHODS

  • The experiments test close-weights models GPT-4o and Gemini 1.5 Pro alongside open-weights models, and examine the impact of query batching.The method also describes many-shot ICL and batched many-shot ICL relative to zero-shot and few-shot prompting.Source: §3 · pp. 3, 5

3.1 MODELS

  • The model set includes three API-access models and two open-weights models; the main text focuses on GPT-4o and Gemini 1.5 Pro because GPT-4o outperforms GPT-4(V)-Turbo.Claude3-Opus is excluded because it accepted at most 20 images per request at the time of the study.Source: §3.1 · p. 5

3.3 EVALUATION METRICS

  • Performance is measured with accuracy, macro-averaged F1, or mean IoU according to task, while data efficiency is estimated by regressing performance on log10(N + 1).The regression is constrained to pass through zero-shot performance and approximates gains expected from an order-of-magnitude increase in demonstrations.Source: §3.3 · pp. 5–6
  • Figure 3 compares Gemini 1.5 Pro with Llama3.2 and InternLM-XComposer2.5, showing gains from demonstrations for Gemini but not for the open-weights models.Source: Figure 3 · p. 6

4 RESULTS

The results show that increasing demonstration examples often improves multimodal ICL, while batching queries can preserve or improve performance and reduce inference costs. Benefits and scaling patterns vary across models, datasets, and settings.

4.1 INCREASING NUMBER OF DEMONSTRATING EXAMPLES

  • Gemini 1.5 Pro improves substantially with more demonstrations on all datasets except DrugOOD Assay; performance continues rising to about 1,000 examples on 8 of 14 datasets.Its largest reported gains over zero-shot are +23% accuracy on HAM10000, +29% on FIVES, and +38% on EuroSAT.Source: §4.1 · p. 6
  • GPT-4o improves substantially except on FIVES and DrugOOD Assay, but its scaling is less consistent, often following V-shaped curves as demonstrations increase.Its shorter context window and greater susceptibility to timeouts with longer inputs limited the number of demonstrations tested relative to Gemini 1.5 Pro.Source: §4.1 · pp. 6–7
  • Llama 3.2-Vision and InternLM-XComposer2.5 do not benefit from demonstration examples, unlike the closed-weight models.The authors identify this as a gap between open- and closed-weight multimodal foundation models.Source: §4.1 · p. 7
  • Across two datasets, prompt changes cause small performance deviations while preserving the overall log-linear improvement trend.The prompt-robustness experiment was conducted on two datasets.Source: §4.1 · p. 7
  • Gemini 1.5 Pro has higher ICL data efficiency than GPT-4o on all datasets except TerraIncognita and DTD; both models average +17% improvement at their optimal demonstration sizes.Table 2 reports zero-shot and optimal-size performance plus ICL data efficiency; it uses macro-averaged F1 for CheXpert and accuracy for other datasets.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 IMPACT OF BATCHING QUERIES

  • With suitable batch sizes, many-shot performance changes little or improves; single-query batching is suboptimal on many datasets in both many-shot and zero-shot settings.The optimal batch size is among the three largest on all datasets except CheXpert and EuroSAT; zero-shot performance rises consistently with batch size on UCMerced and TerraIncognita.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 COST AND LATENCY ANALYSIS

  • In zero-shot batching, per-example latency falls nearly 10x on HAM10000 and 2x on TerraIncognita; in many-shot, latency and cost fall 35x and 10x on HAM10000, and 20x and 45x on TerraIncognita.The zero-shot per-example cost remains similar because adding queries requires no demonstration context.Source: §4.3 · p. 9

5 DISCUSSION

The discussion presents many-shot ICL as a possible way to adapt multimodal foundation models, while identifying limits in task coverage, context capacity, ablation scope, and training-data transparency.

5 DISCUSSION

  • Many-shot ICL may let users adapt private models to new tasks and domains, potentially without fine-tuning, but its absolute performance and data efficiency versus fine-tuning remain untested.The authors also note that it can provide results quickly after model release.Source: §5 · p. 9
  • The study covers selected common multimodal tasks; context limits still constrain many-shot ICL on datasets with several hundred or more classes, and ablations tested only Gemini 1.5 Pro.The authors identify broader tasks and testing whether ablation trends hold for GPT-4o as future work.Source: §5 · p. 9
  • Because private-model training datasets are undisclosed, the authors cannot determine whether their selected datasets appeared in training, despite imperfect zero-shot performance.They state that imperfect zero-shot results suggest—but do not establish—that the datasets were not used for training.Source: §5 · p. 10

6 CONCLUSION

Many-shot ICL brings substantial performance improvements across multiple domains and tasks, especially for Gemini 1.5 Pro, while open-weights models do not show the same benefits. The appendix specifies prompts for classification, batched queries, visual QA, and object localization, and tests prompt wording on two datasets.

6 CONCLUSION

  • Many-shot ICL achieves substantial performance improvements across multiple domains and tasks, especially with Gemini 1.5 Pro, while open-weights models do not show the same benefits.The conclusion identifies the open-weights gap as a future direction.Source: §6 · p. 10

A.1 PROMPT USED FOR IMAGE CLASSIFICATION EXPERIMENTS

  • Classification demonstrations pair each image with a question, class choices, and its answer choice, followed by a fixed answer-and-confidence format.The prompt asks the model to repeat the template for its response.Source: §A.1 · p. 13

A.2 PROMPTS USED FOR IMAGE CLASSIFICATION EXPERIMENTS WITH BATCHING

  • Batched classification prompts enumerate each test image as a separate question and request a numbered answer and confidence score for each.The prompt loops over the test batch and repeats the response template per question.Source: §A.2 · p. 13

A.3.1 PREFIXING IMAGES

  • The prefixing prompt introduces preceding images as examples from the same dataset, then asks each test-image question with class choices.It requests a numbered answer and confidence score for each question.Source: §A.3.1 · p. 14

A.4 PROMPT USED FOR VISUAL QUESTION ANSWERING EXPERIMENTS

  • Visual QA demonstrations pair an image with a question and answer, while test prompts request a numbered answer for each query.The prompt specifies radiology or satellite expertise.Source: §A.4 · p. 14

A.5 PROMPT USED FOR OBJECT LOCALIZATION EXPERIMENTS

  • Object-localization prompts ask for a bounding box around the named object using the format [ymin, xmin, ymax, xmax].Demonstration prompts include the box answer after the requested format.Source: §A.5 · p. 15

B PROMPT SELECTION

  • The prompt-wording robustness experiment randomly selected HAM10000 and EuroSAT because of budget limits.It used a different set of prompts to test robustness to wording differences.Source: B PROMPT SELECTION · p. 15

B.1 PROMPTS USED FOR PROMPT SELECTION EXPERIMENTS

The appendix presents three alternative classification question wordings; prompt 1 is used in the other image-classification experiments.

B.1 PROMPTS USED FOR PROMPT SELECTION EXPERIMENTS

  • Only the question section is shown, and prompt 1 is used for all other image-classification experiments.The appendix presents three question-wording variants.Source: §B.1 · p. 15

B.1.1 PROMPT 1

  • Prompt 1 asks what is in the image and lists the available class choices.Its question and choices follow the image.Source: §B.1.1 · p. 15
  • Prompt 1 places the image before the question and class choices.The question is phrased as “What is in the image above?”Source: §B.1.1 · p. 15

B.1.3 PROMPT 3

  • Prompt 3 places the image after the question and asks the model to classify it.Its wording is “Classify the image above.”Source: §B.1.3 · p. 15
  • Prompt 3 places the question label before the image marker, then asks to classify the image and choose from the class descriptions; prompts 1 and 2 place the image marker before the question.Source: §B.1.3 · p. 15, §B.1.1 · p. 15, §B.1.2 · p. 15

B.2 PROMPT SELECTION RESULTS

Across the two tested datasets, performance shows a consistent log-linear improvement trend across three prompts, despite small deviations.

B.2 PROMPT SELECTION RESULTS

  • Small performance deviations across three prompts leave the overall log-linear improvement trend consistent on the two tested datasets.Figure 6 shows performance as the number of demonstrating examples increases.Source: §B.2 · p. 15

C GPT4(V)-TURBO PERFORMANCE UNDER MANY-SHOT ICL

GPT4(V)-Turbo shows mixed many-shot ICL results: it improves substantially on four classification datasets but has minimal or no gains on six others. Its tested example count was limited by a shorter context window and scaling-related timeout errors.

C GPT4(V)-TURBO PERFORMANCE UNDER MANY-SHOT ICL

  • GPT4(V)-Turbo improves substantially on HAM1000, UCMerced, EuroSAT, and DTD, but shows minimal or no improvement on the other six datasets.Figure 7 compares accuracy as the number of demonstrating examples varies across the 10 classification datasets.Source: C GPT4(V)-TURBO PERFORMANCE UNDER… · p. 15, Figure 7 · p. 16
  • GPT4(V)-Turbo could not reach Gemini 1.5 Pro's demonstration count because its context window is shorter and scaling is more prone to timeout errors.This constrains the comparison of the models at the highest example counts.Source: C GPT4(V)-TURBO PERFORMANCE UNDER… · p. 16
  • GPT4(V)-Turbo generally underperforms Gemini 1.5 Pro, except on FIVES and EuroSAT, where it mostly matches Gemini.The passage describes these comparisons as general rather than universal.Source: C GPT4(V)-TURBO PERFORMANCE UNDER… · p. 16
  • 40 demonstration examples mark peak GPT4(V)-Turbo performance on DrugOOD Assay, where performance has high variance.Figure 7 also shows drops before the best performance at the largest example count on UCMerced, Oxford Pets, and DTD.Source: C GPT4(V)-TURBO PERFORMANCE UNDER… · p. 16, Figure 7 · p. 16

D PERFORMANCE OF MANY-SHOT ICL ON MEDICAL QA TASKS

The medical QA prompt frames the model as an exam expert and presents demonstrations as questions, choices, and answers. It requests a fixed answer-and-confidence format for the actual question.

D PERFORMANCE OF MANY-SHOT ICL ON MEDICAL QA TASKS

  • Demonstrations are formatted as questions with choices and their answers.The prompt introduces the model as an expert in answering medical exam questions.Source: §D.1 · p. 17
  • The actual question and its choices are followed by a requested answer and numerical confidence score from 0 to 1.The format is specified between begin- and end-template markers.Source: §D.1 · p. 17
  • The prompt instructs the model not to deviate from the format and to repeat the template for its answer.Source: §D.1 · p. 17

D.2 RESULTS

Figure 8 reports medical QA results: GPT4(V)-Turbo consistently outperforms Gemini 1.5 Pro, whose performance is more variable. The plotted x-axis uses a log scale.

D.2 RESULTS

Related papers

Loading 2405.09798v2…