PaperlayerView exact revision on arXiv

Source-linked AI summary

Finetune like you pretrain: Improved finetuning of zero-shot vision models

Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, Aditi Raghunathan

arXiv:2212.00638v1cs.CVcs.LG

TL;DR

Finetuning CLIP-like image-text models can behave very differently under small procedural changes, especially across in-distribution and distribution-shifted data. The paper proposes FLYP, which continues contrastive pretraining using class-label text prompts, and reports consistent gains over alternative finetuning methods across distribution-shift, transfer, and few-shot benchmarks. These results support contrastive finetuning as a simple baseline for supervised finetuning of image-text models.

  • Problem

    Small changes to finetuning procedures can substantially change image-text models’ in-distribution and out-of-distribution performance, leaving no simple recipe for the correct modification.

  • Method

    FLYP casts downstream class labels as text prompts and minimizes the same contrastive loss used during image-text pretraining.

  • Results

    FLYP consistently outperforms alternative finetuning approaches across distribution-shift, transfer-learning, and few-shot benchmarks.

  • Takeaways & Limitations

    The results provide evidence for using contrastive finetuning as a simple standard baseline for supervised finetuning of image-text models.

Abstract

from arXiv · show

Finetuning image-text models such as CLIP achieves state-of-the-art accuracies on a variety of benchmarks. However, recent works like WiseFT (Wortsman et al., 2021) and LP-FT (Kumar et al., 2022) have shown that even subtle differences in the finetuning process can lead to surprisingly large differences in the final performance, both for in-distribution (ID) and out-of-distribution (OOD) data. In this work, we show that a natural and simple approach of mimicking contrastive pretraining consistently outperforms alternative finetuning approaches. Specifically, we cast downstream class labels as text prompts and continue optimizing the contrastive loss between image embeddings and class-descriptive prompt embeddings (contrastive finetuning). Our method consistently outperforms baselines across 7 distribution shifts, 6 transfer learning, and 3 few-shot learning benchmarks. On WILDS-iWILDCam, our proposed approach FLYP outperforms the top of the leaderboard by $2.3\%$ ID and $2.7\%$ OOD, giving the highest reported accuracy. Averaged across 7 OOD datasets (2 WILDS and 5 ImageNet associated shifts), FLYP gives gains of $4.2\%$ OOD over standard finetuning and outperforms the current state of the art (LP-FT) by more than $1\%$ both ID and OOD. Similarly, on 3 few-shot learning benchmarks, our approach gives gains up to $4.6\%$ over standard finetuning and $4.4\%$ over the state of the art. In total, these benchmarks establish contrastive finetuning as a simple, intuitive, and state-of-the-art approach for supervised finetuning of image-text models like CLIP. Code is available at https://github.com/locuslab/FLYP.

1 Introduction

Standard cross-entropy finetuning can improve in-distribution performance while reducing robustness to distribution shifts, and subtle modifications have produced inconsistent recipes. FLYP instead continues contrastive pretraining with label-derived text prompts and consistently outperforms prior approaches across settings.

  • Motivation: CLIP-style models use contrastive pretraining to align image and text embeddings, enabling zero-shot classification through class-descriptive prompts.For a new classification problem, images are assigned the class whose text embedding has the highest similarity.
  • Motivation: Standard supervised finetuning and its variants can improve in-distribution performance while affecting robustness to distribution shifts.LP-FT and weight ensembling show that small process changes can alter both in-distribution and out-of-distribution performance.
  • FLYP: FLYP minimizes the pretraining contrastive loss between labeled images and text prompts constructed from their class labels.The method avoids additional multi-stage finetuning or ensembling, although it can also be combined with ensembling.
  • Results: 2.3% ID and 2.7% OOD are FLYP’s gains over ModelSoups on WILDS-iWILDCam using CLIP ViT-L/14@336px.The paper reports the highest ever reported accuracy for this benchmark.
  • Results: 4.2% OOD is FLYP’s gain over full finetuning averaged across 7 OOD datasets, while it exceeds LP-FT by more than 1% both ID and OOD.The datasets comprise 2 WILDS shifts and 5 ImageNet-associated shifts.
  • Interpretation: FLYP’s gains are not explained by prompts or encoder updates alone, and correcting minibatch class overlap slightly decreases performance.Cross-entropy with the same prompts and encoder updates performs worse than FLYP.
  • Conclusion: The paper argues that FLYP’s contribution is empirical evidence that a simple contrastive objective can outperform more complex finetuning methods.The authors propose it as a standard baseline for finetuning zero-shot classifiers.

2 Preliminaries

The preliminaries define CLIP’s joint image-text embedding setup and contrastive pretraining objective, then contrast zero-shot classification with supervised linear probing, full finetuning, LP-FT, and weight ensembling.

  • Task: The image encoder f maps images to a d-dimensional joint embedding space, while the language encoder g maps text descriptions to the same space.The image and language encoders are parameterized by θimg and θtext, respectively.
  • Contrastive pretraining: Contrastive pretraining aligns each image embedding with its paired text embedding and separates it from other text embeddings in the batch.The objective is applied symmetrically to text-to-image comparisons, using normalized embeddings.
  • Zero-shot classification: Zero-shot classification constructs class prompts from templates and predicts the class whose normalized text embedding has the highest similarity to the image embedding.Multiple prompt templates may be ensembled into a zero-shot linear head.
  • Finetuning baselines: Linear probing learns a classifier on frozen image embeddings by minimizing cross-entropy on labeled downstream data.The image encoder remains fixed during this procedure.
  • Finetuning baselines: Full finetuning updates both the classifier and image encoder with downstream cross-entropy, initializing the classifier from zero-shot weights.LP-FT first learns a linear probe, then performs full finetuning from that solution.
  • Finetuning baselines: Weight ensembling linearly interpolates pretrained image-encoder weights with finetuned weights.The method is presented as an alternative for combining zero-shot and finetuned models.

3 FLYP: Finetune like you pretrain

FLYP continues CLIP-style contrastive pretraining during supervised finetuning by pairing labeled images with class-description prompts and updating the encoders with the pretraining objective.

  • FLYP method: FLYP continues pretraining on labeled downstream data by minimizing contrastive loss between image embeddings and text descriptions of target classes.The method constructs image-text pairs from labeled examples and class prompts, then applies stochastic gradient descent to the same pretraining objective.
  • FLYP method: For each label, FLYP samples or selects text descriptions from a set of possible class prompts, such as contextualized class-name templates.The text-description distribution is uniform over possible descriptions for each class.
  • FLYP method: At inference, FLYP predicts the class whose finetuned text embedding has the highest similarity to the finetuned image embedding.This preserves the zero-shot prediction form while using the finetuned encoders.
  • FLYP vs. standard finetuning: Unlike standard CLIP finetuning, FLYP updates both image and language encoders rather than typically updating only the image encoder.This difference reflects FLYP's continuation of the joint image-text pretraining process.
  • FLYP vs. standard finetuning: Ablations show that updating the language encoder or imposing text-embedding structure on a cross-entropy head does not fully explain FLYP's advantage.Cross-entropy variants that update language parameters or use text-derived classifier weights perform worse than FLYP, implicating the loss choice.

4 Experiments

Across distribution-shift, few-shot, and transfer-learning settings, FLYP consistently outperforms established finetuning baselines, with gains in both ID and OOD performance.

  • Experimental setup: FLYP is evaluated across distribution shifts, few-shot learning, and transfer learning using CLIP models and comparisons against standard and improved finetuning baselines.The experiments include ImageNet shifts, WILDS datasets, binary few-shot datasets, and six transfer datasets; baselines include LP, full finetuning, L2-SP, and LP-FT.
  • Distribution shifts: 2.3% ID and 2.7% OOD gains over the leaderboard are reported on WILDS-iWILDCam using ViT-L/14@336px.The comparison is against compute-intensive ModelSoups, which ensembles more than 70 finetuned models.
  • Few-shot learning: FLYP outperforms LP-FT by 1.5% OOD in 4-shot and 0.8% OOD in 16-shot ImageNet classification with weight ensembling.The comparison uses the average OOD accuracy over five ImageNet-shift datasets.

5 Ablations: Why does FLYP improve performance?

The ablations support FLYP’s central hypothesis that matching the contrastive pretraining objective, rather than any individual design change, drives its gains. Updating both encoders and retaining the original contrastive setup matter more than prompt count or collision correction.

  • Ablation design: FLYP combines joint image-language encoder updates, contrastive loss, and prompt sampling, distinguishing it from standard finetuning.These are the three main changes examined in the ablations.
  • Class collisions: FLYP continues to outperform baselines despite class collisions, reaching 99% accuracy on euroSAT, on-par with baselines.It also outperforms baselines on the binary PatchCamelyon and SST2 datasets.
  • Class collisions: 1.3% worse performance on PatchCamelyon results when collision terms are masked, compared with naive FLYP.The authors interpret this as evidence that seemingly beneficial changes can hurt when they depart from pretraining.
  • Prompt templates: The number of text templates does not affect FLYP’s ImageNet ID or OOD accuracy: one template matches 80 templates without ensembling.The experiments also use a single template for PatchCamelyon, SST2, and Flowers.
  • Encoder updates: 2.5% ID and 1.3% OOD drops on iWILDCam occur when the language encoder is frozen during FLYP finetuning.Freezing the language encoder also deteriorates performance on ImageNet.
  • Interpretation: Overall, FLYP’s gains appear to come from exactly matching pretraining, while adding cross-entropy degrades performance.The paper states that no individual change accounts for all gains.

6 Related works

The paper situates FLYP among standard, robust, contrastive, and pretraining-matching approaches to finetuning. Its claimed distinction is that directly matching the pretraining loss outperforms more complex alternatives across multiple settings.

  • Standard finetuning: Linear probing and full finetuning are standard approaches for pretrained supervised, self-supervised, and vision-text models.The paper compares FLYP with these approaches for vision-text models.
  • Robust finetuning: Image-text pretraining improves robustness, motivating extensive work on downstream OOD performance and robust finetuning.The cited literature includes CLIP and several subsequent finetuning methods.
  • Ensembling: Weight averaging of finetuned and zeroshot models improves accuracy and robustness, while large ensembles with LP-FT achieve state-of-the-art results.The paper compares against two-model ensembling on ViT-B/16 for computational reasons.
  • Alternative methods: LP-FT combines linear probing and finetuning and outperforms alternatives based on explicit regularization, selective updates, or smoothness regularizers.Other approaches retain relevant pretraining data or constrain parameter updates.
  • Contrastive learning: Contrastive loss has also been used for supervised learning and language-model finetuning, but those settings differ from FLYP’s zero-shot vision-classifier finetuning.The paper frames FLYP as advocating contrastive finetuning of pretrained image-text models.
  • Matching pretraining: The authors claim FLYP is the first work to document that matching pretraining and finetuning losses outperforms more complex alternatives across several settings and datasets.They relate this general idea to the unified text-text formulation used for T5 models.

7 Conclusion

The conclusion presents FLYP as a straightforward way to finetune zero-shot vision classifiers by retaining the contrastive pretraining objective. Across the paper’s comparisons, the method consistently outperforms alternative finetuning approaches.

  • Method: FLYP finetunes prompt-based classifiers from labeled data with the pretraining contrastive loss instead of typical cross-entropy.This is the method’s basic approach for zero-shot vision classifiers.
  • Findings: FLYP consistently outperforms alternative finetuning approaches developed specifically for image-text models.The conclusion emphasizes this result despite the method’s simplicity.
  • Implication: The authors propose contrastive finetuning as a standard baseline for evaluating finetuning methods for image-text models.They describe the evidence as strong support for adopting this baseline.

A.1 Adding cross-entropy loss to FLYP

The appendix tests whether adding cross-entropy improves FLYP and examines batch-size sensitivity. The reported comparisons favor the original contrastive objective, while smaller batches can reduce OOD accuracy.

  • Adding cross-entropy: FLYP’s weight-ensembling curve completely dominates versions that add cross-entropy under various regularization strengths on ImageNet.The same comparison is also conducted on iWILDCam.
  • Batch size: 1% lower OOD accuracy occurs on ImageNet when batch size decreases from 512 to 256, despite similar ID accuracy.The experiments use batch size 512 for ImageNet and 256 for the other datasets.

A.3 Few-shot classification using CLIP ViT-L/14

FLYP continues to outperform baseline finetuning methods in binary few-shot classification with the larger CLIP ViT-L/14 model. Under 32-shot classification, its gains over LP-FT are reported on both SST2 and PatchCamelyon.

  • A.3 Few-shot classification using CLIP ViT-L/14: FLYP outperforms baselines in binary few-shot classification with CLIP ViT-L/14 on SST2 and PatchCamelyon.The comparison uses the larger CLIP ViT-L/14 model and mirrors an earlier experiment with CLIP ViT-B/16.
  • A.3 Few-shot classification using CLIP ViT-L/14: 4.2%: FLYP outperforms LP-FT on SST2 under 32-shot classification.
  • A.3 Few-shot classification using CLIP ViT-L/14: 3.1%: FLYP outperforms LP-FT on PatchCamelyon under 32-shot classification.

A.4 ImageNet Distribution Shifts - Detailed Results

Detailed ImageNet-associated shift results show that FLYP with weight ensembling consistently outperforms the baselines across all listed distribution shifts.

  • A.4 ImageNet Distribution Shifts - Detailed Results: FLYP with weight ensembling outperforms the baselines across all ImageNet-associated distribution shifts.The same comparison also reports FLYP outperforming baselines on the corresponding in-distribution benchmark.

B Experimental details

The experiments sweep learning rates and weight decay, select hyperparameters using in-distribution validation accuracy, and repeat few-shot sampling 50 times to address variance.

  • B Experimental details: Learning-rate and weight-decay sweeps use five values each for most datasets, while ImageNet uses smaller sweeps because of computational cost.Most datasets use batch size 256; ImageNet uses batch size 512, and L2-SP additionally tunes λ.
  • B Experimental details: Hyperparameters are selected by early stopping on in-distribution validation accuracy, with an 80:20 training-validation split when no public validation split exists.Out-of-distribution data is reserved exclusively for evaluation.
  • B Experimental details: 50 runs: each k-shot setting samples k training and k validation points repeatedly before reporting mean test accuracy.The selected hyperparameter has the lowest mean validation loss across the 50 runs.
Loading 2212.00638v1…