PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners

Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, Peng Gao

arXiv:2303.02151v1cs.CVcs.CL

TL;DR

Few-shot vision must learn from limited annotated data, motivating methods that exploit more diverse pre-training knowledge. CaFo cascades CLIP, DINO, DALL-E, and GPT-3 through prompting, synthetic-data generation, and adaptive caching, achieving state-of-the-art few-shot classification on 11 datasets. Its stated scope remains open to integrating additional pre-trained knowledge.

  • Problem

    Few-shot visual recognition requires generalized representations from limited annotated images, raising whether diverse pre-training knowledge can be cascaded for better few-shot learning.

    Source: §1 · pp. 1–2

  • Method

    CaFo uses GPT-3 to prompt CLIP, DALL-E to generate synthetic training images, and a learnable cache to blend CLIP and DINO predictions.

    Source: §1 · p. 2

  • Results

    CaFo achieves state-of-the-art few-shot classification on 11 datasets without using extra annotated data.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    CaFo demonstrates that four pre-training knowledge types can be unified for few-shot learning through collaboration across foundation models.

    Source: §1 · p. 2, §5 · p. 8

    The paper identifies integrating additional pre-trained knowledge as future work beyond its four-model cascade.

    Source: §5 · p. 9

Abstract

from arXiv · show

Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown promising few-shot performance benefited from the contrastive language-image pre-training. We then question, if the more diverse pre-training knowledge can be cascaded to further assist few-shot representation learning. In this paper, we propose CaFo, a Cascade of Foundation models that incorporates diverse prior knowledge of various pre-training paradigms for better few-shot learning. Our CaFo incorporates CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, DALL-E's vision-generative knowledge, and GPT-3's language-generative knowledge. Specifically, CaFo works by 'Prompt, Generate, then Cache'. Firstly, we leverage GPT-3 to produce textual inputs for prompting CLIP with rich downstream linguistic semantics. Then, we generate synthetic images via DALL-E to expand the few-shot training data without any manpower. At last, we introduce a learnable cache model to adaptively blend the predictions from CLIP and DINO. By such collaboration, CaFo can fully unleash the potential of different pre-training methods and unify them to perform state-of-the-art for few-shot classification. Code is available at https://github.com/ZrrSkywalker/CaFo.

1. Introduction

Few-shot learning addresses vision tasks where models must generalize from limited annotated images. CaFo responds by cascading four foundation-model knowledge types through prompting, generation, and adaptive caching.

  • Few-shot learning targets data-deficient settings where networks learn from limited annotated images.Source: §1 · p. 1
  • CaFo asks whether pre-learned knowledge from multiple self-supervisory paradigms can be adaptively integrated for stronger few-shot learning.Source: §1 · p. 2
  • CaFo combines CLIP, DINO, DALL-E, and GPT-3 within a “Prompt, Generate, then Cache” pipeline.The models contribute language-contrastive, vision-contrastive, vision-generative, and language-generative knowledge.Source: §1 · p. 2
  • GPT-3 produces semantic prompts, DALL-E expands few-shot data, and a cache model adaptively ensembles CLIP and DINO predictions.Source: §1 · p. 2
  • CaFo achieves state-of-the-art few-shot classification on 11 datasets without extra annotated data.Source: §1 · p. 2

2. Related Work

Related work develops transferable vision representations through supervised or self-supervised pre-training, language-assisted prompting, and few-shot adaptation methods. CaFo builds on these directions by using language-generative knowledge for semantic prompts.

Pre-training of Vision Models

  • Vision models commonly pre-train on large datasets and fine-tune downstream, while self-supervised learning reduces dependence on annotations.Source: §2 · p. 2

Language-assisted Vision Models

  • Language-assisted vision methods show that prompt format affects task accuracy, motivating learnable or language-model-generated textual inputs.Source: §2 · p. 2

Few-shot Learning

  • Few-shot learning includes metric-learning approaches that measure instance-to-category distances and meta-learning approaches that optimize rapidly adaptable initialization.Source: §2 · p. 3

3. Cascade of Foundation Models

CaFo cascades four pre-training paradigms through a “Prompt, Generate, then Cache” pipeline for few-shot learning. GPT-3 enriches CLIP prompts, DALL-E expands training data, and a cache model adaptively combines CLIP and DINO predictions.

Different Pre-training Paradigms

  • CaFo combines CLIP’s language-contrastive, DINO’s vision-contrastive, GPT-3’s language-generative, and DALL-E’s vision-generative knowledge.The method is designed to integrate prior knowledge from multiple pre-training paradigms.Source: §3 · p. 3, §3.1 · p. 3

Prompt

  • GPT-3 generates semantically rich prompts from category-specific language commands, which are then fed into CLIP’s textual encoder.Commands can be customized for specialized downstream categories, such as pet descriptions in OxfordPets.Source: §3.1 · p. 3, §3.2 · p. 4

Generate

  • DALL-E generates synthetic category images, while CLIP filters the top-K′ images to expand each class’s few-shot training set.K′ is kept comparable with K to preserve the low-data regime, and the zero-shot expansion requires no collection or annotation manpower.Source: §3.2 · p. 4

Cache

  • The cache model stores CLIP and DINO visual features as two key types with shared one-hot label values for adaptive knowledge ensemble.Only the cached adapter keys are learnable during training; the pre-trained CLIP and DINO models remain frozen.Source: §3.2 · p. 4

Adaptive Inference

  • At inference, CLIP and DINO features query the cache to produce logits whose ensemble weights are based on distribution similarity to CLIP’s zero-shot prediction.The similarity-based ensemble can suppress obviously false categories and amplify moderately correct ones.Source: §3.3 · p. 5

4. Experiments

CaFo is evaluated across 11 datasets, few-shot settings, distribution shifts, efficiency conditions, and ablations of its cascaded components. It consistently achieves leading performance, while experiments clarify how synthetic-data quantity, adaptive inference, and backbone choice affect results.

Performance

  • CaFo surpasses existing methods across different ImageNet shot settings, with 1-shot CaFo outperforming 8-shot Linear-probe CLIP and CoOp, and 8-shot CaFo exceeding all 16-shot methods.CaFo also achieves 68.79% accuracy with only 10 minutes of training under the reported efficiency setting.Source: §4.2 · p. 6, Figure 5 · p. 6, Table 1 · p. 6
  • CaFo shows leading performance across 10 additional datasets and remains comparable to 4-shot methods in zero-shot settings on datasets including Caltech101 and OxfordPets.The evaluated domains include real-world scenes, detailed textures, and satellite-captured landscapes.Source: §4.2 · p. 6, Figure 6 · p. 7

Distribution Shift

  • CaFo achieves the best reported out-of-distribution performance on ImageNet-V2 and ImageNet-Sketch, surpassing Tip-Adapter-F by +3.28%, +0.88%, and +3.43%, respectively.Models are trained on ImageNet and tested on the two target datasets.Source: §4.2 · p. 6, Table 3 · p. 6

Ablation Study

  • The full cascade performs best across all tested ImageNet shot counts, while cache-based adaptive ensembling improves as shot number increases and DALL-E and GPT-3 independently boost CLIP and DINO.The ablation compares CLIP, DINO, their adaptive ensemble, DALL-E, GPT-3, and the final CaFo configuration.Source: §4.3 · p. 7, Table 4 · p. 7
  • Larger numbers of generated DALL-E images can reduce performance because top-K′ selection admits more low-quality images, so K′ remains comparable to the original K shots.This preserves the few-shot regime while expanding training data without manual collection or annotation.Source: §4.3 · p. 7, Table 6 · p. 8
  • Naive average or maximum pooling degrades accuracy without adaptive weights, whereas the distribution-similarity ensemble performs best when combined with pZS.pZS can suppress wrong predictions from other logits and amplify moderately correct ones.Source: §4.3 · pp. 7–8, Table 5 · p. 7
  • CaFo retains leading performance across different CLIP visual backbones, indicating generalizability to network architectures.The visual-encoder comparison is conducted on 16-shot ImageNet.Source: §4.3 · p. 8, Table 7 · p. 8

Visualization

  • DALL-E-generated images highlight target-category semantics and expand few-shot training data, while GPT-3 prompts provide richer descriptions that better depict visual appearances and support correct classification.The visualizations cover ImageNet, OxfordPets, and Caltech101 for DALL-E, and ImageNet for GPT-3 prompting.Source: §4.4 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8

5. Conclusion

CaFo cascades diverse knowledge from multiple foundation models through a “Prompt, Generate, then Cache” pipeline for few-shot learning. It achieves state-of-the-art few-shot performance on 11 datasets, while motivating future integration of additional pre-trained knowledge.

5. Conclusion

  • CaFo combines GPT-3 prompting, DALL-E data generation, and adaptive DINO–CLIP fusion in a unified “Prompt, Generate, then Cache” pipeline.The method incorporates diverse knowledge from different pre-training paradigms.Source: §5 · p. 8
  • CaFo achieves state-of-the-art few-shot learning performance on 11 datasets.Source: §5 · p. 9
  • Future work will integrate additional pre-trained knowledge, including masked-generative, 3D-contrastive, and 3D-generative models.Source: §5 · p. 9

A. Additional Performance Comparison

CaFo remains stronger than Tip-Adapter-F when DALL-E-generated images and GPT-3-created prompts are removed, across the compared datasets.

A. Additional Performance Comparison

B. Additional Ablation Study

Additional ablations examine alternative cache-model foundation models, zero-shot synthetic-data augmentation, and the cache sharpness hyperparameter. CLIP+DINO performs best among tested cache combinations, while zero-shot gains and β sensitivity vary by setting.

B. Additional Ablation Study

  • CLIP+DINO, the final cache-model solution, performs best among the tested foundation-model combinations.The alternatives include SimCLR, MAE, and SLIP; SLIP can improve accuracy as an enhanced version of CLIP.Source: B. Additional Ablation Study · p. 13, Table 8 · p. 13
  • +4.62% and +7.54% are the zero-shot CaFo gains over CLIP on Caltech101 and EuroSAT, respectively.The best number of DALL-E-generated images varies from 1∼16 across datasets, and larger numbers do not generally improve results.Source: B. Additional Ablation Study · p. 13
  • 0.6 performs best for β on 16-shot ImageNet.β controls the sharpness of the cache affinity matrix through the non-linear modulator ϕ(x) = exp(−β ·(1−x)).Source: B. Additional Ablation Study · p. 13, Table 9 · p. 13

C. Additional Visualization

Additional visualizations examine category separation, learning dynamics, GPT-3 prompts, and DALL-E images. Across these views, CaFo is reported to improve cluster contrast, convergence, accuracy, prompt-based correction, and synthetic-data expansion.

C. Additional Visualization

Related papers

Loading 2303.02151v1…