Source-linked AI summary
ECO: Ensembling Context Optimization for Vision-Language Models
Lorenzo Agnolucci, Alberto Baldrati, Francesco Todino, Federico Becattini, Marco Bertini, Alberto Del Bimbo
TL;DR
Prompt wording affects downstream classification, and existing learned-prompt methods leave prompt ensembling unexplored. ECO learns an ensemble of shorter contexts and reports consistent improvements across 11 benchmarks, with no additional inference-time overhead.
Problem
Prompt wording strongly influences downstream performance, while existing learned-prompt methods learn only a single prompt rather than exploiting prompt ensembling.
Source: §1 · p. 1Method
ECO learns multiple prompts with reduced context tokens, combines their features by averaging, and can be used with existing prompt-learning strategies.
Source: §1 · p. 2, §2.2 · pp. 2–3Results
ECO reports consistent improvements over competing methods across 11 benchmarks, with the largest gains at 1 or 2 shots.
Source: §4 · p. 5Takeaways & Limitations
Balancing context length and prompt count is beneficial for few-shot image classification, particularly with 1 or 2 shots.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Image recognition has recently witnessed a paradigm shift, where vision-language models are now used to perform few-shot classification based on textual prompts. Among these, the CLIP model has shown remarkable capabilities for zero-shot transfer by matching an image and a custom textual prompt in its latent space. This has paved the way for several works that focus on engineering or learning textual contexts for maximizing CLIP's classification capabilities. In this paper, we follow this trend by learning an ensemble of prompts for image classification. We show that learning diverse and possibly shorter contexts improves considerably and consistently the results rather than relying on a single trainable prompt. In particular, we report better few-shot capabilities with no additional cost at inference time. We demonstrate the capabilities of our approach on 11 different benchmarks.
1. Introduction
CLIP's downstream classification depends strongly on prompt wording, while existing learned-prompt methods use a single prompt. ECO combines prompt learning with prompt ensembling and reports improvements over competing methods on 11 testing datasets without additional inference overhead.
1. Introduction
- Prompt wording strongly affects downstream performance, making hand-crafted prompt selection non-trivial.The paper gives a Caltech101 example in which adding “a” to a prompt reportedly increases accuracy by 5%.Source: §1 · p. 1
- Prompt ensembling averages textual features from different prompts and is used to improve robustness and results [15].Source: §1 · p. 2
- Prior learned methods improve on hand-crafted prompts but learn only a single prompt, leaving learned prompt ensembling unexplored.The cited methods are CoOp [21], CoCoOp [20], and MaPLe [10].Source: §1 · p. 2
- ECO learns multiple prompts with reduced context tokens and combines them through prompt ensembling.Figure 1 contrasts D prompts with N context tokens each against CoOp's single prompt with M tokens, with M = D × N.Source: Figure 1 · p. 1, §1 · p. 2
- ECO reports significant improvements over competing methods on 11 testing datasets and adds no inference-time computational overhead.The approach is described as applicable on top of potentially any existing prompt-learning technique.Source: §1 · p. 2
2. Method
CLIP classifies images by comparing image and text features in a shared embedding space. ECO learns multiple context-token sets, averages their text features, and keeps the pretrained CLIP model frozen during training.
2.1. Preliminaries
- CLIP zero-shot classification compares an image feature with text-prompt features for candidate classes to compute class probabilities.The probability uses cosine similarity scaled by CLIP's learned temperature parameter.Source: §2.1 · p. 2
2.2. ECO
- ECO learns multiple sets of context tokens and averages their encoded text features across prompts for each class.The same context vectors are shared among classes, as in CoOp.Source: §2.2 · p. 2
- ECO's prompt-wise average combines learned prompt-specific features, joining prompt learning with prompt ensembling.The paper contrasts these data-learned contexts with hand-created prompts and relates the averaging approach to prompt ensembling [15].Source: §2.2 · p. 3
- Training updates context vectors through cross-entropy while CLIP remains frozen; after training, averaged text encodings can be precomputed without extra inference computation.ECO sets N and D so their product matches CoOp's M context vectors; D = 1 and N = M coincides with CoOp.Source: §2.2 · p. 3
3. Experimental Results
ECO is evaluated against prompt-learning baselines across 11 image-classification benchmarks, with multiple prompt and context-length settings tested under few-shot protocols. The results favor prompt ensembles, especially at low shot counts.
3.1. Evaluation Protocol
- The evaluation follows 1-, 2-, 4-, 8-, and 16-shot training, averages three seeds, and compares parameter-matched ECO variants with CoOp.CoOp is represented by ECO with D=1 and N=16; the authors use CoOp as the basic comparison and identify CoCoOp and MaPLe as future extensions.Source: §3.1 · p. 3, §3 · p. 3
- The evaluation covers ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101.Performance is evaluated on the full test sets.Source: §3.1 · p. 3
3.2. Quantitative Results
- Across 11 benchmarks, ECO consistently outperforms competing methods, with absolute improvements up to 3.47 over CoOp and the largest gains at 1–2 shots.The comparison reports average accuracy across the benchmarks and varies the number of shots.Source: Table 1 · p. 3, §3.2 · pp. 3–4
- With N=4 and D=4, ECO achieves the best average performance; it also beats zero-shot CLIP on noisy-label Food101.Figure 2 reports results across the test datasets while varying shots, prompts, and context tokens.Source: §3.2 · p. 3, Figure 2 · p. 4
4. Conclusion
The paper concludes that optimizing an ensemble of contexts improves results consistently across 11 benchmarks. It highlights the value of balancing context length and prompt count, particularly with very few shots.
4. Conclusion
- ECO optimizes an ensemble of multiple contexts and yields consistent improvements across 11 benchmarks.Source: §4 · p. 5
- The strategy is applicable on top of potentially any existing prompt-learning technique with no additional inference overhead.Source: §4 · p. 5
- Balancing context length and prompt count is beneficial for CLIP few-shot classification, with the biggest gains at 1 or 2 shots.Source: §4 · p. 5