PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes Good Examples for Visual In-Context Learning?

Yuanhan Zhang, Kaiyang Zhou, Ziwei Liu

arXiv:2301.13670v2cs.CV

TL;DR

Visual in-context learning can handle unseen tasks without parameter updates, but its performance depends strongly on the chosen examples. This paper studies that sensitivity and introduces unsupervised and supervised prompt retrieval methods, which improve over random selection, with supervised retrieval performing best in the reported experiments.

  • Problem

    Visual in-context learning lacks comprehensive evidence on how to select examples, despite performance being highly sensitive to prompt choice.

  • Method

    The paper proposes prompt retrieval that scores source examples and offers unsupervised nearest-example search plus supervised neural selection targeting in-context learning performance.

  • Results

    Prompt retrieval clearly outperforms random selection across foreground segmentation, single object detection, and colorization, with supervised retrieval performing best.

  • Takeaways & Limitations

    Good visual prompts should balance semantic similarity with contextual similarity, including viewpoint, background, and appearance.

  • Takeaways & Limitations

    The methods are not strong enough under distribution shifts, where their advantage over random selection is much smaller than on a standard benchmark.

Abstract

from arXiv · show

Large-scale models trained on broad data have recently become the mainstream architecture in computer vision due to their strong generalization performance. In this paper, the main focus is on an emergent ability in large vision models, known as in-context learning, which allows inference on unseen tasks by conditioning on in-context examples (a.k.a.~prompt) without updating the model parameters. This concept has been well-known in natural language processing but has only been studied very recently for large vision models. We for the first time provide a comprehensive investigation on the impact of in-context examples in computer vision, and find that the performance is highly sensitive to the choice of in-context examples. To overcome the problem, we propose a prompt retrieval framework to automate the selection of in-context examples. Specifically, we present (1) an unsupervised prompt retrieval method based on nearest example search using an off-the-shelf model, and (2) a supervised prompt retrieval method, which trains a neural network to choose examples that directly maximize in-context learning performance. The results demonstrate that our methods can bring non-trivial improvements to visual in-context learning in comparison to the commonly-used random selection.

1. Introduction

Large vision models enable in-context learning for unseen tasks without parameter updates, but performance is highly sensitive to the selected examples. The paper proposes prompt retrieval methods to automate selection and reports that supervised retrieval performs best across several visual tasks.

  • Large-scale vision models are pretrained on broad data and demonstrate strong generalization, but their size and proprietary training data often limit users to Model-as-a-Service APIs.
  • In-context learning adapts large vision models to unseen tasks by prepending domain-specific input-output examples to a test example without updating model parameters.The examples are also called prompts.
  • Performance is highly sensitive to in-context example choice, with closer examples sometimes producing better results because of similar object pose and background.A mismatched example can cause the predicted mask to focus on the wrong region.
  • The prompt retrieval framework scores source instances for suitability and selects the highest-scoring examples to construct a prompt.
  • Unsupervised retrieval uses nearest-example search with an off-the-shelf model, whereas supervised retrieval trains a neural network to select examples that maximize in-context learning performance.Both methods interpret the score as cosine distance measuring query-example similarity.
  • Across foreground segmentation, single object detection, and image colorization, supervised prompt retrieval performs best and often finds examples that are semantically close and spatially similar to the query.

2. Methods

The paper formulates visual in-context learning as prediction conditioned on a prompt of labeled examples, then proposes automatic prompt retrieval because random example selection can produce large performance variation. It compares unsupervised nearest-example search with supervised retrieval trained from in-context performance.

  • Visual In-Context Learning: Visual in-context learning conditions a query prediction on K input-output pairs in a prompt without updating the large model’s parameters.The prompt contains examples from the dataset, while the model produces the query output.
  • Prompt Design: Random prompt selection can make performance highly sensitive to which in-context examples are chosen, with the best-to-worst gap exceeding 70% mIoU.The common approach randomly selects within-class image-label pairs.
  • Prompt Retrieval: Prompt retrieval scores each source instance for a query and selects the highest-scoring example or top-K example pairs to construct the prompt.The scoring function fθ compares a source example and query, implemented with feature extraction and cosine distance.
  • Unsupervised Prompt Retrieval: The unsupervised method uses a fixed off-the-shelf feature extractor to retrieve examples by cosine distance between query and training-example features.Its scoring parameters remain fixed rather than being optimized for in-context learning.
  • Supervised Prompt Retrieval: The supervised method makes the feature extractor learnable and directly optimizes prompt selection for in-context learning through a contrastive-learning objective.It computes each candidate’s in-context prediction performance, then forms positive and negative sets from the top-5 highest- and lowest-performing examples.
  • Supervised Prompt Retrieval: During contrastive training, each mini-batch example is paired with sampled positive and negative examples, with the negative set also including other examples from the mini-batch.Positive and negative representations are compared using cosine distance.

3. Experiments

Experiments evaluate prompt retrieval against random selection across three unseen vision tasks, distribution shifts, backbones, and design choices. Retrieval generally improves performance, with supervised retrieval strongest, while gains vary by task and shift.

  • Setup: Experiments compare Random, UnsupPR, and SupPR on foreground segmentation, single object detection, and colorization benchmarks.The evaluated tasks were unseen during training of the image inpainting model.
  • Main Results: Prompt retrieval clearly outperforms random selection, with SupPR achieving the best performance overall.The main results cover all three benchmarks.
  • Distribution Shifts: Under Pascal-to-MSCOCO distribution shift, SupPR remains stronger than UnsupPR and Random, but its average gain is around 3% rather than 8%.The reported averages are 19.95% vs. 16.78% under shift and 35.56% vs. 27.56% on the standard benchmark.
  • Backbones: Performance is broadly insensitive to the choice among CLIP, EVA, and ViT backbones.These backbones use multimodal contrastive, self-supervised, and supervised pre-training, respectively.
  • Further Analysis: SupPR retrieves examples more similar to queries in semantics, background, pose, appearance, and viewpoint than UnsupPR.The qualitative analysis reports similar patterns across other categories and tasks.
  • Further Analysis: Larger retrieval sets benefit both prompt retrieval methods, whereas random selection is unaffected and retrieval performance eventually plateaus.For SupPR, 20% of the total data is sufficient for decent performance.
  • Further Analysis: Changing the order of three selected examples generally has little effect, while adding more examples improves all three methods.Distance metrics also perform similarly, according to the cited ablation.

4. Related Work

Visual in-context learning extends the in-context paradigm from language to vision, where recent work has explored large vision models and downstream task customization. This paper focuses on the underexplored problem of finding visual examples that improve downstream performance.

  • In-context learning: In-context learning conditions inference on task-specific input-output pairs without changing model parameters.The paradigm lets users customize outputs for unseen tasks while model parameters remain inaccessible or unchanged.
  • Vision research: Early vision research includes Flamingo for image-video processing and a pure vision model trained to fill missing patches in figures and infographics.The paper distinguishes its focus from these earlier efforts by studying example selection.
  • Research focus: This work studies how to find good visual in-context examples that benefit downstream performance.The paper follows prior pure-vision work but examines example selection rather than the same modeling dimension.
  • Related findings: Natural-language studies show that example choice, prompt length, and example order can substantially affect in-context performance.These findings motivated research on retrieving effective examples for language models.
  • Related findings: Prior language methods selected semantically nearest examples or combined unsupervised candidate retrieval with supervised ranking for downstream performance.These approaches provide precedents for the prompt-retrieval methods developed for visual in-context learning.

5. Discussion and Conclusion

The paper finds that visual in-context performance varies sharply with example choice and proposes prompt retrieval methods to address this sensitivity. It further identifies semantic and contextual similarity as important, while noting weaker performance under distribution shifts.

  • Discussion and Conclusion: Different in-context examples can produce drastically different downstream results, motivating automated prompt retrieval.The paper reports significant improvements over random selection across varied problem settings.
  • Discussion and Conclusion: Prompt retrieval is implemented through unsupervised and supervised methods, with the latter learning to choose examples that maximize in-context performance.The framework supports selecting examples using learned or off-the-shelf similarity signals.
  • Discussion and Conclusion: Good examples should be semantically and contextually similar to the query in viewpoint, background, and appearance.The paper argues that models balancing spatial and semantic closeness would better support visual example retrieval than semantics-focused models alone.
  • Discussion and Conclusion: Under distribution shifts, the methods outperform random selection but achieve a much smaller gap than on a standard benchmark.The authors identify substantial room for improvement in handling shifted distributions.

A. Illustration of In-context Examples

The supplementary material provides additional visual in-context learning results across foreground segmentation, single object detection, and colorization.

  • A. Illustration of In-context Examples: Supplementary figures illustrate further results for foreground segmentation, single object detection, and colorization.These examples extend the reported visual in-context learning analyses beyond the main paper.

A.1. Foreground Segmentation

The supplementary material extends the foreground-segmentation examples beyond the person and cow categories shown in the main paper.

  • A.1. Foreground Segmentation: Figures 6–11 present examples from the remaining 18 Pascal-5i categories.The main paper presents in-context examples from the person and cow categories.

A.2. Single Object Detection

For single object detection, SupPR retrieves in-context examples that more closely match queries in object pose and viewpoint than UnsupPR.

  • SupPR-selected examples are more similar to detection queries in object pose than UnsupPR-selected examples.
  • SupPR-selected examples are more similar to detection queries in viewpoint than UnsupPR-selected examples.
  • The comparison is illustrated with examples including Fig. 12(f) for pose and Fig. 12(r) for viewpoint.

A.3. Coloralization

The visualizations compare prompts retrieved by UnsupPR and SupPR across colorization and related task examples. In colorization, SupPR retrieves examples whose ground-truth images more closely match query image style, including background color.

  • Colorization: Colorization maps a gray-scale image to a color image, with query images shown alongside their ground truth.
  • Colorization: SupPR retrieves colorization examples whose ground-truth images more closely match query image style, including background color.
  • Foreground segmentation: Foreground-segmentation visualizations cover train, tv, bus, bottle, sheep, bird, boat, airplane, bicycle, car, cat, chair, dog, horse, motorbike, table, plant, and sofa categories.
  • Single object detection: Single-object-detection visualizations compare UnsupPR- and SupPR-retrieved examples, including differences in object pose and viewpoint.
Loading 2301.13670v2…