PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners

Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, Peng Gao

arXiv:2303.02151v1cs.CVcs.CL

TL;DR

Few-shot vision must learn from limited annotated data, motivating methods that exploit more diverse pre-training knowledge. CaFo cascades CLIP, DINO, DALL-E, and GPT-3 through prompting, synthetic-data generation, and adaptive caching, achieving state-of-the-art few-shot classification on 11 datasets. Its stated scope remains open to integrating additional pre-trained knowledge.

  • Problem

    Few-shot visual recognition requires generalized representations from limited annotated images, raising whether diverse pre-training knowledge can be cascaded for better few-shot learning.

  • Method

    CaFo uses GPT-3 to prompt CLIP, DALL-E to generate synthetic training images, and a learnable cache to blend CLIP and DINO predictions.

  • Results

    CaFo achieves state-of-the-art few-shot classification on 11 datasets without using extra annotated data.

  • Takeaways & Limitations

    CaFo demonstrates that four pre-training knowledge types can be unified for few-shot learning through collaboration across foundation models.

  • Takeaways & Limitations

    The paper identifies integrating additional pre-trained knowledge as future work beyond its four-model cascade.

Abstract

from arXiv · show

Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown promising few-shot performance benefited from the contrastive language-image pre-training. We then question, if the more diverse pre-training knowledge can be cascaded to further assist few-shot representation learning. In this paper, we propose CaFo, a Cascade of Foundation models that incorporates diverse prior knowledge of various pre-training paradigms for better few-shot learning. Our CaFo incorporates CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, DALL-E's vision-generative knowledge, and GPT-3's language-generative knowledge. Specifically, CaFo works by 'Prompt, Generate, then Cache'. Firstly, we leverage GPT-3 to produce textual inputs for prompting CLIP with rich downstream linguistic semantics. Then, we generate synthetic images via DALL-E to expand the few-shot training data without any manpower. At last, we introduce a learnable cache model to adaptively blend the predictions from CLIP and DINO. By such collaboration, CaFo can fully unleash the potential of different pre-training methods and unify them to perform state-of-the-art for few-shot classification. Code is available at https://github.com/ZrrSkywalker/CaFo.

1. Introduction

Few-shot learning addresses vision tasks where models must generalize from limited annotated images. CaFo responds by cascading four foundation-model knowledge types through prompting, generation, and adaptive caching.

  • Few-shot learning targets data-deficient settings where networks learn from limited annotated images.
  • CaFo asks whether pre-learned knowledge from multiple self-supervisory paradigms can be adaptively integrated for stronger few-shot learning.
  • CaFo combines CLIP, DINO, DALL-E, and GPT-3 within a “Prompt, Generate, then Cache” pipeline.The models contribute language-contrastive, vision-contrastive, vision-generative, and language-generative knowledge.
  • GPT-3 produces semantic prompts, DALL-E expands few-shot data, and a cache model adaptively ensembles CLIP and DINO predictions.
  • CaFo achieves state-of-the-art few-shot classification on 11 datasets without extra annotated data.

2. Related Work

Related work develops transferable vision representations through supervised or self-supervised pre-training, language-assisted prompting, and few-shot adaptation methods. CaFo builds on these directions by using language-generative knowledge for semantic prompts.

  • Pre-training of Vision Models: Vision models commonly pre-train on large datasets and fine-tune downstream, while self-supervised learning reduces dependence on annotations.
  • Language-assisted Vision Models: Language-assisted vision methods show that prompt format affects task accuracy, motivating learnable or language-model-generated textual inputs.
  • Few-shot Learning: Few-shot learning includes metric-learning approaches that measure instance-to-category distances and meta-learning approaches that optimize rapidly adaptable initialization.

3. Cascade of Foundation Models

CaFo cascades four pre-training paradigms through a “Prompt, Generate, then Cache” pipeline for few-shot learning. GPT-3 enriches CLIP prompts, DALL-E expands training data, and a cache model adaptively combines CLIP and DINO predictions.

  • Different Pre-training Paradigms: CaFo combines CLIP’s language-contrastive, DINO’s vision-contrastive, GPT-3’s language-generative, and DALL-E’s vision-generative knowledge.The method is designed to integrate prior knowledge from multiple pre-training paradigms.
  • Prompt: GPT-3 generates semantically rich prompts from category-specific language commands, which are then fed into CLIP’s textual encoder.Commands can be customized for specialized downstream categories, such as pet descriptions in OxfordPets.
  • Generate: DALL-E generates synthetic category images, while CLIP filters the top-K′ images to expand each class’s few-shot training set.K′ is kept comparable with K to preserve the low-data regime, and the zero-shot expansion requires no collection or annotation manpower.
  • Cache: The cache model stores CLIP and DINO visual features as two key types with shared one-hot label values for adaptive knowledge ensemble.Only the cached adapter keys are learnable during training; the pre-trained CLIP and DINO models remain frozen.
  • Adaptive Inference: At inference, CLIP and DINO features query the cache to produce logits whose ensemble weights are based on distribution similarity to CLIP’s zero-shot prediction.The similarity-based ensemble can suppress obviously false categories and amplify moderately correct ones.

4. Experiments

CaFo is evaluated across 11 datasets, few-shot settings, distribution shifts, efficiency conditions, and ablations of its cascaded components. It consistently achieves leading performance, while experiments clarify how synthetic-data quantity, adaptive inference, and backbone choice affect results.

  • Performance: CaFo surpasses existing methods across different ImageNet shot settings, with 1-shot CaFo outperforming 8-shot Linear-probe CLIP and CoOp, and 8-shot CaFo exceeding all 16-shot methods.CaFo also achieves 68.79% accuracy with only 10 minutes of training under the reported efficiency setting.
  • Performance: CaFo shows leading performance across 10 additional datasets and remains comparable to 4-shot methods in zero-shot settings on datasets including Caltech101 and OxfordPets.The evaluated domains include real-world scenes, detailed textures, and satellite-captured landscapes.
  • Distribution Shift: CaFo achieves the best reported out-of-distribution performance on ImageNet-V2 and ImageNet-Sketch, surpassing Tip-Adapter-F by +3.28%, +0.88%, and +3.43%, respectively.Models are trained on ImageNet and tested on the two target datasets.
  • Ablation Study: The full cascade performs best across all tested ImageNet shot counts, while cache-based adaptive ensembling improves as shot number increases and DALL-E and GPT-3 independently boost CLIP and DINO.The ablation compares CLIP, DINO, their adaptive ensemble, DALL-E, GPT-3, and the final CaFo configuration.
  • Ablation Study: Larger numbers of generated DALL-E images can reduce performance because top-K′ selection admits more low-quality images, so K′ remains comparable to the original K shots.This preserves the few-shot regime while expanding training data without manual collection or annotation.
  • Ablation Study: Naive average or maximum pooling degrades accuracy without adaptive weights, whereas the distribution-similarity ensemble performs best when combined with pZS.pZS can suppress wrong predictions from other logits and amplify moderately correct ones.
  • Ablation Study: CaFo retains leading performance across different CLIP visual backbones, indicating generalizability to network architectures.The visual-encoder comparison is conducted on 16-shot ImageNet.
  • Visualization: DALL-E-generated images highlight target-category semantics and expand few-shot training data, while GPT-3 prompts provide richer descriptions that better depict visual appearances and support correct classification.The visualizations cover ImageNet, OxfordPets, and Caltech101 for DALL-E, and ImageNet for GPT-3 prompting.

5. Conclusion

CaFo cascades diverse knowledge from multiple foundation models through a “Prompt, Generate, then Cache” pipeline for few-shot learning. It achieves state-of-the-art few-shot performance on 11 datasets, while motivating future integration of additional pre-trained knowledge.

  • 5. Conclusion: CaFo combines GPT-3 prompting, DALL-E data generation, and adaptive DINO–CLIP fusion in a unified “Prompt, Generate, then Cache” pipeline.The method incorporates diverse knowledge from different pre-training paradigms.
  • 5. Conclusion: CaFo achieves state-of-the-art few-shot learning performance on 11 datasets.
  • 5. Conclusion: Future work will integrate additional pre-trained knowledge, including masked-generative, 3D-contrastive, and 3D-generative models.

A. Additional Performance Comparison

CaFo remains stronger than Tip-Adapter-F when DALL-E-generated images and GPT-3-created prompts are removed, across the compared datasets.

  • A. Additional Performance Comparison: CaFo without DALL-E-generated images or GPT-3-created prompts consistently outperforms the second-best Tip-Adapter-F on 10 datasets.

B. Additional Ablation Study

Additional ablations examine alternative cache-model foundation models, zero-shot synthetic-data augmentation, and the cache sharpness hyperparameter. CLIP+DINO performs best among tested cache combinations, while zero-shot gains and β sensitivity vary by setting.

  • B. Additional Ablation Study: CLIP+DINO, the final cache-model solution, performs best among the tested foundation-model combinations.The alternatives include SimCLR, MAE, and SLIP; SLIP can improve accuracy as an enhanced version of CLIP.
  • B. Additional Ablation Study: +4.62% and +7.54% are the zero-shot CaFo gains over CLIP on Caltech101 and EuroSAT, respectively.The best number of DALL-E-generated images varies from 1∼16 across datasets, and larger numbers do not generally improve results.
  • B. Additional Ablation Study: 0.6 performs best for β on 16-shot ImageNet.β controls the sharpness of the cache affinity matrix through the non-linear modulator ϕ(x) = exp(−β ·(1−x)).

C. Additional Visualization

Additional visualizations examine category separation, learning dynamics, GPT-3 prompts, and DALL-E images. Across these views, CaFo is reported to improve cluster contrast, convergence, accuracy, prompt-based correction, and synthetic-data expansion.

  • C. Additional Visualization: GPT-3-produced prompts assist CaFo in rectifying false predictions from CLIP’s original templates.Additional examples are visualized for ImageNet.
  • C. Additional Visualization: DALL-E-generated images highlight target-category semantics and expand few-shot training sets in low-data regimes.Examples span ImageNet, OxfordPets, and Caltech101.
  • C. Additional Visualization: CaFo shows more contrastive category clusters than Tip-Adapter-F and mitigates aliasing between similar classes.
  • C. Additional Visualization: Collaborating DALL-E, DINO, and GPT-3 with CLIP improves convergence speed and test accuracy on 16-shot ImageNet.The comparison uses 20-epoch test-accuracy learning curves.
  • C. Additional Visualization: CaFo shows state-of-the-art performance across all few-shot settings on 10 datasets in the reported comparison.The comparison also identifies CaFo without DALL-E images and GPT-3 prompts as “CaFo w/o D.&G.”
Loading 2303.02151v1…