PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving CLIP Training with Language Rewrites

Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, Yonglong Tian

arXiv:2305.20088v2cs.CVcs.CLcs.LG

TL;DR

CLIP applies augmentation only to images, leaving text unchanged and limiting language diversity during training. LaCLIP uses LLM-generated rewrites as text augmentations and improves transfer performance across datasets, with no added training computation or memory.

  • Problem

    CLIP augments images but leaves texts unchanged, increasing text overfitting risk and motivating effective sentence-level language augmentation that preserves key concepts and meanings.

  • Method

    LaCLIP uses LLaMA in-context learning to generate diverse caption rewrites, then randomly samples the original caption or a rewrite during CLIP training.

  • Results

    LaCLIP significantly improves CLIP embedding transferability across pretraining datasets and scales; on LAION-400M, ImageNet zero-shot accuracy rises from 62.0% to 64.4%.

  • Takeaways & Limitations

    Language rewriting provides a simple, scalable, and versatile text-augmentation strategy that improves vision-language model performance without additional training-time memory or computation.

  • Takeaways & Limitations

    Generating rewrites can require significant GPU resources, and unfiltered LLaMA outputs may include image-misaligned details that affect downstream transferability.

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.

1 Introduction

CLIP applies augmentation only to images, leaving texts unchanged and limiting language-side diversity. LaCLIP addresses this asymmetry with LLM-generated language rewrites that preserve key concepts while improving transfer performance.

  • CLIP’s augmentation asymmetry leaves text inputs unchanged while images are augmented each epoch, limiting language-side supervision.
  • Repeated exposure to identical captions increases text overfitting risk and significantly affects zero-shot transferability.
  • Existing word-level replacement and masking methods weakly enrich sentence structure and do not preserve concepts through effective language rewriting.
  • LaCLIP uses LLaMA in-context learning to generate diverse caption variants from meta-input-output examples while preserving original concepts and meanings.
  • 2.4 percentage points: LaCLIP raises LAION-400M ImageNet zero-shot accuracy from 62.0% to 64.4% over CLIP.

2 Related Works

Related work establishes CLIP’s contrastive vision-language foundation and surveys text augmentation methods, while noting efficiency costs in approaches that add training inputs or losses.

  • CLIP learns visual representations and zero-shot transfer ability by applying contrastive learning to image and text features.
  • ALIGN follows a related vision-language approach and achieves similar performance using larger and noisier datasets.
  • Some follow-up vision-language methods add training inputs and losses, potentially reducing training efficiency and increasing memory consumption.
  • Earlier text augmentation methods include synonym replacement, random masking, and back translation.

3 Improving CLIP with Language Rewrites

LaCLIP augments CLIP’s unchanged text inputs with LLM-generated rewrites. It uses in-context examples to produce diverse captions, then randomly trains on the original or rewritten text without added training-time computation or parameters.

  • Language Rewrites as Text Augmentation: LaCLIP generates diverse caption variants with LLM in-context learning to enhance CLIP training.
  • Language Rewrites as Text Augmentation: Standard CLIP augments images but leaves text unchanged, whereas LaCLIP introduces a text augmentation function augT for rewritten captions.
  • Large-scale Language Rewriting: In-context learning conditions LLaMA on a few meta-input-output examples before rewriting a target image description.
  • Meta-Input-Output Text Pair Generation: ChatGPT, Bard, COCO, and Human strategies provide four types of meta-input-output pairs, with 16 pairs generated for each strategy.
  • Large-scale Language Rewriting: LLaMA-7B generates four rewrites for every dataset caption, corresponding to the four meta-input-output sources.
  • LaCLIP: Training CLIP with Language Augmentations: LaCLIP randomly samples the original caption or one of M = 4 rewrites as text augmentation during training.
  • LaCLIP: Training CLIP with Language Augmentations: LaCLIP adds no computation or parameter overhead during training because all other CLIP components remain unchanged.

4 Experiments

Experiments evaluate LaCLIP across datasets, transfer settings, augmentation strategies, meta-input sources, and backbone sizes. Language rewrites improve transfer performance broadly while preserving the training computation and memory cost of CLIP.

  • Few-Shot & Linear-Probing: Language rewrites almost always outperform vanilla CLIP or SLIP in 5-way, 5-shot transfer and linear probing.The comparison covers ImageNet and 15 common downstream datasets for linear probing.
  • Zero-shot Evaluation: LaCLIP improves ImageNet and downstream-dataset zero-shot performance across all pretrained datasets, including over 8% absolute top-1 ImageNet improvement on CC12M.On CC12M, it also improves average accuracy on other downstream datasets by 7%.
  • Zero-shot Evaluation: LaCLIP integrates with other CLIP-based methods without additional training computation or memory overhead after augmented texts are generated.Applying the strategy to SLIP produces LaSLIP and improves performance across all evaluation metrics.
  • Ablation Studies: LLM-based text augmentation scales better with increasing numbers of augments than simpler strategies because it provides greater diversity.Figure 3 treats zero augments as vanilla CLIP and compares ImageNet zero-shot accuracy as augmentation count varies.
  • Ablation Studies: Different meta-input-output sources yield similar performance, while human-generated prompts may benefit from image access for more accurate and diverse rewrites.The tested sources are ChatGPT, Bard, COCO, and human rewrites.

5 Multi-Text Training Loss with LaCLIP

LaCLIP-MT extends language-rewrite training by pairing each image with all available rewritten texts in a multi-positive contrastive objective. This objective further improves performance over LaCLIP across most metrics on CC12M and RedCaps.

  • Multi-Text Training Loss with LaCLIP: LaCLIP-MT pairs each image with its original caption and all rewritten versions to form multi-positive training pairs.This approach incurs a slight computational cost compared with standard LaCLIP.
  • Multi-Text Training Loss with LaCLIP: The final LaCLIP-MT loss averages the image loss over multi-positive pairs with the text loss: L = (LI∗ + LT)/2.The text-side loss iterates over all texts, with each text still paired with a single image.
  • Multi-Text Training Loss with LaCLIP: LaCLIP-MT further improves on LaCLIP across most metrics on CC12M and RedCaps.The comparison includes vanilla CLIP, LaCLIP, and LaCLIP-MT.

6 Conclusion, Limitations and Broader Impact

LaCLIP augments CLIP training with LLM-generated text rewrites and improves CLIP embeddings across pre-training scales and datasets. The approach has generation-cost, rewrite-quality, factuality, and environmental limitations.

  • Conclusion: LaCLIP incorporates text augmentations through LLM-based rewriting and improves CLIP embedding performance across varied pre-training scales and datasets.The approach uses in-context learning capabilities of LLMs and also introduces a multi-text training loss.
  • Limitations: Generating text rewrites can require significant GPU resources and hours for large datasets, despite no additional memory or computation overhead during training.The limitation concerns preprocessing rewrite generation rather than the CLIP training process itself.
  • Limitations: Unfiltered LLaMA rewrites may add image-misaligned details that can affect downstream transferability.The authors suggest filtering rewritten texts to retain relevant and accurate versions.
  • Broader Impact: LLM-generated rewrites may inherit factual errors and hallucinations from large-scale web training data, motivating additional filtering before real-world deployment.The rewriting strategy also requires substantial GPU/TPU computation, which can contribute to a higher carbon footprint.

Appendices

The appendices provide supplementary experiment details, results, and analysis.

  • Appendices: Supplementary materials contain additional experiment details, results, and analysis.

A.1 Pre-training Datasets

The study trains CLIP and LaCLIP on four image-text pre-training datasets spanning different scales and sources, with dataset versions affected by image link rot. Images are resized to a 256-pixel shorter side, and evaluation uses ImageNet plus 15 downstream datasets.

  • Pre-training Datasets: The experiments train CLIP and LaCLIP on CC3M, CC12M, RedCaps, and LAION-400M, while also training SLIP and LaSLIP on CC12M.The acquired dataset versions may contain slightly fewer images than their original versions because of image link rot.
  • Pre-training Datasets: CC3M contains 3.3 million image-text pairs from 5 billion webpages, with the acquired version comprising 2.8 million unique samples.
  • Pre-training Datasets: CC12M contains 12.4 million image-text pairs with relaxed filters and broader topics; the acquired version contains 10.0 million samples.
  • Pre-training Datasets: RedCaps contains 12.0 million Reddit image-caption pairs from 350 subreddits, with 11.7 million unique samples in the acquired version.
  • Pre-training Datasets: LAION-400M contains 413 million unique samples in its original version and 340 million samples in the acquired version after Common Crawl processing and filtering.
  • Pre-training Datasets: All images are resized so that their shorter side measures 256 pixels, and evaluation covers ImageNet plus 15 widely used downstream datasets.

B. Implementation Details

The appendices specify encoder, training, evaluation, prompt, and rewrite-template configurations, alongside qualitative comparisons of augmentation strategies. These details cover the main CLIP variants and LaCLIP meta-input-output sources.

  • Encoders: The experiments use ViT-S/16, ViT-B/16, ViT-B/32, and ViT-L/16 vision encoders, with the smallest CLIP text encoder and a 49,408-word vocabulary.The text encoder has a maximum context length of 77.
  • Hyper-Parameters: Image pre-training uses RandomResizedCrop, and the process runs on four machines with eight A100 GPUs each.SLIP uses a learning rate of 3 × 10^-3 and weight decay of 0.1.
  • Zero-shot Classification: Zero-shot evaluation averages text embeddings across prompt templates before computing image-to-class similarities, using 80 prompts for LAION-400M ImageNet models.Models trained on other datasets use seven templates to expedite evaluation.
  • Implementation Details: The supplementary tables report encoder details, pre-training and SLIP hyperparameters, ImageNet linear-probing settings, and CC12M strategy comparisons.
  • Few-shot Classification: Few-shot evaluation measures 5-way 5-shot performance across 15 downstream datasets using Prototypical Networks on frozen vision features without data augmentation.Each model is evaluated over 600 randomly sampled episodes.
  • Linear-Probing: ImageNet linear probing freezes the image encoder, trains a linear classifier, applies RandomHorizontalFlip, and reports the best result across a swept learning-rate range.
  • C.1 Meta-input-output Pairs: The rewrite templates use four meta-input-output strategies—ChatGPT, Bard, Human, and MSCOCO—with 16 pairs generated or selected for each strategy.The pairs serve as examples in the LLaMA in-context learning framework.
  • D.1 Augmentation Qualitative Comparison: Qualitative comparisons show that EDA and back translation mainly modify words, whereas LLM augmentation changes sentence structure while preserving key concepts and meaning.

D.2 Detailed Experiment Results on Augmentation Strategy

The LLM-based augmentation strategy consistently outperforms EDA and back translation, and LaCLIP’s performance scales better as more rewrites are added. Feature visualizations further show clearer class boundaries and more distinct clusters for LaCLIP.

  • Augmentation strategies: Four augmented texts per original sentence were generated for EDA, back translation, and LLaMA-based augmentation under a fair comparison.EDA applies predefined operations, back translation uses four languages, and LLaMA ICL uses four meta-input-output pairs.
  • Augmentation strategies: LLM-based augmentation consistently achieves superior results across datasets and evaluation metrics, outperforming EDA and back translation.The baseline strategies improve vanilla CLIP, but the LLM-based strategy performs better overall.
  • Scaling with augmentation count: Baseline augmentation strategies do not scale well as the number of augmentations per sentence increases, indicating limited diversity in their rewritten texts.The comparison evaluates zero to four augmentations per original text on CC12M.
  • Scaling with augmentation count: LaCLIP scales well with more augmentations because LLaMA ICL produces rich and diverse rewrites, enabling continued performance improvement.This contrasts with the limited scaling observed for the baseline strategies.
  • Feature visualization: LaCLIP produces clearer class boundaries and more distinct clusters than other approaches across CIFAR-10, Food101, STL-10, and EuroSAT.The t-SNE visualizations suggest improved image embedding spaces for downstream tasks.

G. Detailed Experiment Results for LaCLIP-MT

LaCLIP-MT extends LaCLIP by pairing each image with all corresponding texts, providing more diverse supervision during training. Its gains are especially pronounced on RedCaps.

  • Evaluation: The LaCLIP, LaCLIP-MT, and CLIP comparison includes zero-shot and linear-probing experiments on CC12M and RedCaps.The experiment is summarized in Table A9.
  • Multi-text training: LaCLIP-MT pairs each image with all corresponding texts, allowing the vision encoder to receive more diverse supervision during training iterations.The comparison covers pre-training on CC12M and RedCaps.
  • Results: On RedCaps, LaCLIP-MT achieves an additional 1.9% increase in zero-shot classification accuracy on ImageNet.The reported comparison is among CLIP, LaCLIP, and LaCLIP-MT.

I. Ablation on LLaMA model

The LLaMA ablation examines model size and sampling temperature for LaCLIP text rewriting. A lightweight 7B model is sufficient, while performance peaks near τ = 0.9.

  • Ablation setup: The ablation studies vary LLaMA model size and temperature using ChatGPT meta-input-output prompting pairs on CC12M.The experiments use a single text augmentation strategy.
  • Model size: A 7B LLaMA model is sufficient to significantly boost LaCLIP performance, while larger models provide only modest overall benefits in these experiments.Larger models improve some downstream datasets, but their aggregate impact is relatively modest.
  • Temperature: Higher sampling temperatures increase text diversity, but excessively high values can introduce random words or non-English tokens that harm results.Temperature controls the balance between diversity and precision during LLaMA generation.
  • Temperature: Performance is robust across temperatures from 0.3 to 1.1 and generally peaks around τ = 0.9.The authors consistently use τ = 0.9 for text rewriting in their experiments.

J. Ablation on Non-contrastive Training

Language rewrites improve performance beyond contrastive CLIP training, including in Virtex, and LaCLIP’s gains appear tied to generalization rather than optimization. LaCLIP also helps distinguish fine-grained ImageNet categories and remains effective with different text-encoder setups.

  • Non-contrastive training: Language augmentation improves La-Virtex over standard Virtex on PASCAL VOC07 linear classification after CC12M pre-training.This result suggests language rewrites may benefit non-contrastive vision-language training methods.
  • Training dynamics: LaCLIP achieves higher validation accuracy and higher training loss than CLIP across CC3M, CC12M, RedCaps, and LAION-400M.The authors interpret this pattern as improved generalization rather than improved optimization.
  • Qualitative analysis: LaCLIP distinguishes some fine-grained ImageNet categories where vanilla CLIP faces challenges.Figure A3 shows examples from the three categories with the largest accuracy improvements.
  • Text-encoder setups: LaCLIP outperforms CLIP models using the evaluated pre-trained BERT text-encoder setups, while freezing the text encoder reduces vanilla CLIP performance.The comparison is conducted on CC12M with fine-tuned or frozen BERT-Base text encoders.
Loading 2305.20088v2…