PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Multimodal Models as General In-Context Classifiers

Marco Garosi, Matteo Farina, Alessandro Conti, Massimiliano Mancini, Elisa Ricci

arXiv:2602.23229v1cs.CV

TL;DR

The paper asks whether LMMs are genuinely worse classifiers than contrastive VLMs or are merely inadequately conditioned. It benchmarks in-context learning for closed- and open-world classification and introduces CIRCLE, which iteratively refines pseudo-labels. LMMs can match or surpass VLM counterparts, while CIRCLE robustly improves open-world classification despite naive ICL’s sensitivity to context.

  • Problem

    Prior studies generally favor contrastive VLMs for classification, while the classification ability of LMMs under in-context conditioning remains insufficiently established.

  • Method

    The paper benchmarks LMM in-context learning in closed- and open-world classification and proposes CIRCLE, a training-free method that iteratively refines pseudo-labels for unlabeled context images.

  • Results

    LMMs with annotated in-context examples can match or surpass contrastive VLMs in closed-world classification, while CIRCLE consistently surpasses VLMs in open-world classification.

  • Takeaways & Limitations

    LMMs are a flexible alternative to specialized classifiers when their context is used effectively, with targeted refinement needed for robust open-world performance.

  • Takeaways & Limitations

    CIRCLE’s lack of human supervision may yield semantically coherent but task-misaligned label interpretations.

Abstract

from arXiv · show

Which multimodal model should we use for classification? Previous studies suggest that the answer lies in CLIP-like contrastive Vision-Language Models (VLMs), due to their remarkable performance in zero-shot classification. In contrast, Large Multimodal Models (LMM) are more suitable for complex tasks. In this work, we argue that this answer overlooks an important capability of LMMs: in-context learning. We benchmark state-of-the-art LMMs on diverse datasets for closed-world classification and find that, although their zero-shot performance is lower than CLIP's, LMMs with a few in-context examples can match or even surpass contrastive VLMs with cache-based adapters, their "in-context" equivalent. We extend this analysis to the open-world setting, where the generative nature of LMMs makes them more suitable for the task. In this challenging scenario, LMMs struggle whenever provided with imperfect context information. To address this issue, we propose CIRCLE, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself. Through extensive experiments, we show that CIRCLE establishes a robust baseline for open-world classification, surpassing VLM counterparts and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.

1. Introduction

Prior work finds contrastive VLMs highly competitive for classification, raising whether LMMs are intrinsically weaker or simply improperly conditioned. This paper studies in-context learning and shows that context can substantially improve LMM classification, including through CIRCLE for open-world settings.

  • VLMs consistently compete strongly with generative LMMs in closed- and open-world classification, motivating whether LMMs are inherently weaker or improperly conditioned.
  • In-context learning lets LMMs perform new tasks without parameter updates by conditioning generation on a few input-output examples.
  • LMMs conditioned on annotated examples gain substantially in closed-world classification and can close or surpass their gap with contrastive VLMs.
  • CIRCLE iteratively refines pseudo-labels for unlabeled in-context images, addressing missing fixed labels and human supervision in open-world classification.
  • An extensive benchmark reports that naive ICL struggles in open-world classification, whereas CIRCLE improves the base model and consistently surpasses VLMs.

2. Related work

Related work positions contrastive VLMs as strong zero-shot classifiers and LMMs as broadly capable but generally weaker classification models. Visual in-context learning studies instead emphasize the importance of example selection for downstream performance.

  • Contrastive VLMs classify by comparing image and text embeddings in a shared space, enabling zero-shot prediction from class names.
  • Prior evaluations generally find generative LMMs underperform contrastive models across closed-world and open-world image-classification settings.
  • In-context learning emerged as a strategy for adapting large multimodal models to specific tasks without changing their large parameter sets.
  • Visual ICL studies show that example selection can significantly affect downstream performance, while other work proposes supervised and unsupervised selection strategies.

3. Closed-world classification

This section formalizes closed-world classification for contrastive VLMs and generative LMMs, then compares cache-based VLM adaptation with LMM in-context learning across few-shot settings.

  • Preliminaries: Closed-world classification selects a label from a finite, known set of semantic categories.
  • Preliminaries: Contrastive VLMs classify images by selecting the natural-language class whose text embedding has the highest cosine similarity with the image embedding.For CLIP models, this corresponds to zero-shot classification through aligned visual and text encoders.
  • Preliminaries: LMMs generate free-form text from image tokens and a textual query, so closed-world classification reformulates the task as a multiple-choice question and parses the response.Predictions are typically extracted through exact or fuzzy string matching against the class options.
  • Few-Shot and In-Context Learning: Few-shot VLM adaptation refines logits using similarity to labeled context images, whereas LMMs condition generation on image-label pairs placed in the context window.Tip-Adapter requires context samples for every predefined category; Vanilla ICL uses ordered image-label examples before the query image and class question.
  • Experimental Setup: The experiments compare Tip-Adapter for VLMs with Vanilla ICL for LMMs at k ∈ {4, 8, 16}, using Textual Inclusion as the evaluation metric.For CLIP models, Textual Inclusion is equivalent to top-1 accuracy.
  • Results: Across ten datasets, increasing context lets LMMs close the gap with contrastive VLMs despite lower zero-shot performance.Phi-3.5-Vision and Qwen2-VL-7B achieve relative zero-shot boosts of up to +29.2% and +17.7% on average, respectively; Qwen2-VL-7B matches CLIP-ViT-L/14 at k=16.

4. Open-world classification

Open-world classification has no predefined class set, making generative LMMs a natural fit but leaving unlabeled context examples to be pseudo-labeled. CIRCLE iteratively refines those pseudo-labels using the other context samples, and consistently outperforms naïve ICL and VLM counterparts.

  • Open-world classification requires image responses without a predefined set of classes, unlike closed-world classification.
  • LMMs are naturally suited to open-world queries because they can answer directly without reformulating them as multiple-choice questions.
  • Pseudo In-Context Learning assigns generated pseudo-labels to unlabeled context images and uses their tokenized representations as context.
  • CIRCLE recursively refines each context image’s pseudo-label using all other in-context samples, addressing inter-sample dependencies absent from naïve pseudo-labeling.The images remain fixed while their pseudo-labels evolve across refinement rounds.
  • CIRCLE’s pipeline pseudo-labels context images, repeatedly refines their labels using the others, and then classifies the query image.
  • Naïve ICL can degrade zero-shot open-world performance, whereas CIRCLE improves inclusion and semantic relevance simultaneously without the same trade-off.Adding context examples helps semantic-related metrics while LI remains stable; additional refinement rounds help over pseudo-labeling but show diminishing returns.
  • CIRCLE consistently outperforms base LMMs, other ICL variants, and VLM counterparts across correctness and relevance metrics and dataset categories.On prototypical tasks, Qwen2-VL reaches 91.5 LI with CIRCLE versus 81.1 for Pseudo ICL; CIRCLE also improves Zero-Shot Phi-3.5-V by +31.4%.

5. Conclusions

The paper finds that LMMs are data-efficient in closed-world classification through ICL but fragile to noisy context, especially in open-world settings. CIRCLE iteratively refines pseudo-labels to stabilize predictions and outperform baselines, while remaining limited by possible task misalignment and streaming overhead.

  • Open-world settings intensify LMM sensitivity to context noise, causing standard ICL to produce inconsistent semantic interpretations.
  • CIRCLE is a training-free self-refinement mechanism that models dependencies across unlabeled in-context examples.
  • CIRCLE stabilizes LMM outputs and yields high-precision predictions that outperform all baselines in open-world classification.
  • Limitations: Without human annotations, refinement may converge to semantically coherent but task-misaligned label interpretations, while streaming updates can add computational overhead.

Supplementary Material

The supplementary material adds technical details, extended experiments, and analyses that complement the main paper. It covers datasets, granular closed-world results, expanded open-world evaluations, and streaming experiments.

  • The supplement provides additional technical and implementation details, extended experimental results, and further analyses.
  • Section A: Section A details the datasets used in the experiments.
  • Section B: Section B extends the main closed-world analysis with per-category performance breakdowns.
  • Section C: Section C reports the full open-world results, including 4-shot and 8-shot configurations.
  • Section D: Section D provides complete data for the streaming experiments.

A. Datasets

The experiments use established training and test splits from previous work, with the evaluation datasets summarized in Table 3.

  • The evaluation datasets are summarized in Table 3.
  • Experiments use the same training and test splits as previous work.

B. Closed-world results

The closed-world supplement reports average results across ten datasets for contrastive VLMs and generative LMMs under multiple few-shot context strategies. Adapter-based VLM methods consistently help, whereas LMM performance depends strongly on context quality, with similarity retrieval producing gains and random context causing severe degradation.

  • Results are averaged across ten datasets, with deltas reported relative to each corresponding zero-shot model.
  • Contrastive VLMs: Contrastive VLM results compare zero-shot models, Tip-Adapter, and k-NN at 4, 8, and 16 shots.
  • Generative LMMs: Generative LMM results compare vanilla, random-context, and similarity-based-context settings at 4, 8, and 16 shots.
  • Contrastive VLMs: Tip-Adapter consistently outperforms the zero-shot baseline, reaching a peak improvement of +8.8% with ViT-B/16 at 16 shots.
  • Contrastive VLMs: k-NN often underperforms zero-shot at 4 shots, including a -8.4% delta for ViT-B/32, but recovers with 16 shots.
  • Generative LMMs: Random context causes performance degradation of up to -48.7% for Qwen-2-VL, whereas similarity retrieval yields +29.2% for Phi-3.5-Vision and +17.7% for Qwen-2-VL.

C. Open-world results

Across five models and ten datasets, CIRCLE consistently improves open-world classification over zero-shot, random-context, and pseudo-label alternatives. Its gains are especially pronounced for fine-grained tasks and larger relevant contexts.

  • Main results: CIRCLE consistently outperforms zero-shot and other ICL variants across the evaluated open-world datasets and model families.The experiments compare Random Context, Pseudo ICL, and CIRCLE at 4, 8, and 16 shots.
  • Main results: With Qwen2.5-VL at 16 shots, CIRCLE raises Prototypical-dataset SS from 47.9 to 67.7, mCS from 31.1 to 67.2, and LI from 82.9 to 94.9.
  • Context quality: 16-shot Random Context can degrade performance sharply, lowering LLaVa OneVision SS from 56.2 to 29.3, while CIRCLE increases it to 74.0.The same comparison reports CIRCLE recovery for mCS and LI as well.
  • Fine-grained classification: CIRCLE delivers consistent Very fine-grained gains, including Phi-3.5-Vision LI increasing from 54.2 to 99.6 and Qwen2.5-VL LI from 69.0 to 93.6.
  • Shot scaling: Increasing relevant context from 4 to 16 shots widens CIRCLE’s advantage, with Qwen2-VL Fine-grained bCS rising from 62.9 to 66.4 at 4 shots and reaching 72.0 at 16 shots.The reported 16-shot scores also reach 61.1 SS and 57.3 mCS.

D. Streaming results

In streaming open-world experiments, CIRCLE remains robust across model families and dataset groups. It generally surpasses zero-shot and pseudo-ICL baselines across multiple semantic and concept-based metrics.

  • Main results: CIRCLE consistently outperforms Zero-Shot and Pseudo ICL across the majority of streaming models and dataset groups.
  • Llama Inclusion: On Prototypical datasets, CIRCLE raises Qwen2-VL LI from 78.7 to 90.4 and Phi-4-Multimodal LI from 49.8 to 84.9.
  • Similarity metrics: For Qwen2-VL on Prototypical datasets, CIRCLE increases SS from 51.9 to 60.9 and mCS from 43.7 to 59.5.
  • Baseline comparison: Pseudo ICL is occasionally strong on individual metrics but remains inconsistent across models, metrics, and datasets, unlike CIRCLE’s overall stability.

E. Implementation details

The experiments evaluate multiple VLM and LMM families under controlled generation and hardware settings. Larger contexts require reduced batch sizes, downscaled images, and sometimes multiple GPUs.

  • Models: The study evaluates three CLIP variants and five LMMs from the Qwen, LLaVa OneVision, and Phi model series.
  • Resource handling: For CIRCLE contexts, batch size is reduced to 8, 4, or 2 samples per GPU depending on the LMM, and context images are downscaled to 224 × 224 pixels.
  • Reproducibility: Experiments use greedy decoding for up to 64 generated tokens and run on NVIDIA A100 GPUs with 40, 64, or 80 GB of VRAM.
  • Evaluation cost: Evaluation time ranges from minutes for zero-shot closed-world experiments to 8–10 hours for the largest streaming datasets.

F. Qualitative results

Qualitative examples across the evaluated datasets show that CIRCLE produces more consistent, appropriately specific labels than baseline methods. Random contexts especially lead to misclassification or unrelated hallucinated categories.

  • Evaluation setup: The qualitative evaluation presents three examples per dataset across a subset of ten datasets, comparing Vanilla, Random Context, Pseudo-labeling, and CIRCLE.
  • Baseline behavior: Baseline methods inconsistently identify the correct concept and sometimes hallucinate unrelated categories, especially with Random Context.
  • CIRCLE behavior: CIRCLE predicts the correct label more consistently than the baselines in the qualitative examples.
  • Mechanism: CIRCLE uses iteratively refined context to guide the LMM toward the correct specificity and output format.
Loading 2602.23229v1…