Improving CLIP Training with Language Rewrites

• CLIP’s augmentation asymmetry leaves text inputs unchanged while images are augmented each epoch, limiting language-side supervision.Source: §1 · pp. 1–2
• Repeated exposure to identical captions increases text overfitting risk and significantly affects zero-shot transferability.Source: §1 · p. 2
• Existing word-level replacement and masking methods weakly enrich sentence structure and do not preserve concepts through effective language rewriting.Source: §1 · p. 2
• LaCLIP uses LLaMA in-context learning to generate diverse caption variants from meta-input-output examples while preserving original concepts and meanings.Source: §1 · p. 2
• 2.4 percentage points: LaCLIP raises LAION-400M ImageNet zero-shot accuracy from 62.0% to 64.4% over CLIP.Source: §1 · p. 2

• CLIP learns visual representations and zero-shot transfer ability by applying contrastive learning to image and text features.Source: §2 · p. 2
• ALIGN follows a related vision-language approach and achieves similar performance using larger and noisier datasets.Source: §2 · p. 2
• Some follow-up vision-language methods add training inputs and losses, potentially reducing training efficiency and increasing memory consumption.Source: §2 · p. 3
• Earlier text augmentation methods include synonym replacement, random masking, and back translation.Source: §2 · p. 3

Language Rewrites as Text Augmentation

• LaCLIP generates diverse caption variants with LLM in-context learning to enhance CLIP training.Source: §3 · p. 3, §3.1 · p. 3
• Standard CLIP augments images but leaves text unchanged, whereas LaCLIP introduces a text augmentation function augT for rewritten captions.Source: §3.1 · p. 3

Large-scale Language Rewriting

• In-context learning conditions LLaMA on a few meta-input-output examples before rewriting a target image description.Source: §3.2 · pp. 3–4, §3.3 · pp. 4–5

Meta-Input-Output Text Pair Generation

• ChatGPT, Bard, COCO, and Human strategies provide four types of meta-input-output pairs, with 16 pairs generated for each strategy.Source: §3.2 · p. 4

Large-scale Language Rewriting

• LLaMA-7B generates four rewrites for every dataset caption, corresponding to the four meta-input-output sources.Source: §3.3 · pp. 4–5

LaCLIP: Training CLIP with Language Augmentations

• LaCLIP randomly samples the original caption or one of M = 4 rewrites as text augmentation during training.Source: §3.4 · p. 5
• LaCLIP adds no computation or parameter overhead during training because all other CLIP components remain unchanged.Source: §3.4 · p. 5

Few-Shot & Linear-Probing

• Language rewrites almost always outperform vanilla CLIP or SLIP in 5-way, 5-shot transfer and linear probing.The comparison covers ImageNet and 15 common downstream datasets for linear probing.Source: Table 2 · p. 7, §4.2 · p. 7, Table 3 · p. 8

Zero-shot Evaluation

• LaCLIP improves ImageNet and downstream-dataset zero-shot performance across all pretrained datasets, including over 8% absolute top-1 ImageNet improvement on CC12M.On CC12M, it also improves average accuracy on other downstream datasets by 7%.Source: §4.1 · pp. 6–7
• LaCLIP integrates with other CLIP-based methods without additional training computation or memory overhead after augmented texts are generated.Applying the strategy to SLIP produces LaSLIP and improves performance across all evaluation metrics.Source: §4.1 · p. 7

Ablation Studies

• LLM-based text augmentation scales better with increasing numbers of augments than simpler strategies because it provides greater diversity.Figure 3 treats zero augments as vanilla CLIP and compares ImageNet zero-shot accuracy as augmentation count varies.Source: §4.3 · p. 8, Figure 3 · p. 9
• Different meta-input-output sources yield similar performance, while human-generated prompts may benefit from image access for more accurate and diverse rewrites.The tested sources are ChatGPT, Bard, COCO, and human rewrites.Source: §4.3 · pp. 8–9

Multi-Text Training Loss with LaCLIP

• LaCLIP-MT pairs each image with its original caption and all rewritten versions to form multi-positive training pairs.This approach incurs a slight computational cost compared with standard LaCLIP.Source: §5 · pp. 9–10
• The final LaCLIP-MT loss averages the image loss over multi-positive pairs with the text loss: L = (LI∗ + LT)/2.The text-side loss iterates over all texts, with each text still paired with a single image.Source: §5 · p. 10
• LaCLIP-MT further improves on LaCLIP across most metrics on CC12M and RedCaps.The comparison includes vanilla CLIP, LaCLIP, and LaCLIP-MT.Source: §5 · p. 10

Conclusion

• LaCLIP incorporates text augmentations through LLM-based rewriting and improves CLIP embedding performance across varied pre-training scales and datasets.The approach uses in-context learning capabilities of LLMs and also introduces a multi-text training loss.Source: §6 · p. 10

Limitations

• Generating text rewrites can require significant GPU resources and hours for large datasets, despite no additional memory or computation overhead during training.The limitation concerns preprocessing rewrite generation rather than the CLIP training process itself.Source: §6 · p. 10
• Unfiltered LLaMA rewrites may add image-misaligned details that can affect downstream transferability.The authors suggest filtering rewritten texts to retain relevant and accurate versions.Source: §6 · p. 10

Broader Impact

• LLM-generated rewrites may inherit factual errors and hallucinations from large-scale web training data, motivating additional filtering before real-world deployment.The rewriting strategy also requires substantial GPU/TPU computation, which can contribute to a higher carbon footprint.Source: §6 · p. 10

Appendices

• Supplementary materials contain additional experiment details, results, and analysis.Source: Appendices · p. 15

Pre-training Datasets

• The experiments train CLIP and LaCLIP on CC3M, CC12M, RedCaps, and LAION-400M, while also training SLIP and LaSLIP on CC12M.The acquired dataset versions may contain slightly fewer images than their original versions because of image link rot.Source: §A.1 · p. 15
• CC3M contains 3.3 million image-text pairs from 5 billion webpages, with the acquired version comprising 2.8 million unique samples.Source: §A.1 · p. 15
• CC12M contains 12.4 million image-text pairs with relaxed filters and broader topics; the acquired version contains 10.0 million samples.Source: §A.1 · p. 15
• RedCaps contains 12.0 million Reddit image-caption pairs from 350 subreddits, with 11.7 million unique samples in the acquired version.Source: §A.1 · p. 15
• LAION-400M contains 413 million unique samples in its original version and 340 million samples in the acquired version after Common Crawl processing and filtering.Source: §A.1 · p. 15
• All images are resized so that their shorter side measures 256 pixels, and evaluation covers ImageNet plus 15 widely used downstream datasets.Source: §A.1 · p. 15, §A.2 · p. 15

Encoders

• The experiments use ViT-S/16, ViT-B/16, ViT-B/32, and ViT-L/16 vision encoders, with the smallest CLIP text encoder and a 49,408-word vocabulary.The text encoder has a maximum context length of 77.Source: B. Implementation Details · p. 15

Hyper-Parameters

• Image pre-training uses RandomResizedCrop, and the process runs on four machines with eight A100 GPUs each.SLIP uses a learning rate of 3 × 10^-3 and weight decay of 0.1.Source: B. Implementation Details · p. 15

Zero-shot Classification

• Zero-shot evaluation averages text embeddings across prompt templates before computing image-to-class similarities, using 80 prompts for LAION-400M ImageNet models.Models trained on other datasets use seven templates to expedite evaluation.Source: B. Implementation Details · pp. 15–16

Implementation Details

• The supplementary tables report encoder details, pre-training and SLIP hyperparameters, ImageNet linear-probing settings, and CC12M strategy comparisons.Source: Table A2 · p. 16, Table A3 · p. 17, Table A4 · p. 17

Few-shot Classification

• Few-shot evaluation measures 5-way 5-shot performance across 15 downstream datasets using Prototypical Networks on frozen vision features without data augmentation.Each model is evaluated over 600 randomly sampled episodes.Source: B. Implementation Details · p. 16

Linear-Probing

• ImageNet linear probing freezes the image encoder, trains a linear classifier, applies RandomHorizontalFlip, and reports the best result across a swept learning-rate range.Source: B. Implementation Details · p. 16

C.1 Meta-input-output Pairs

• The rewrite templates use four meta-input-output strategies—ChatGPT, Bard, Human, and MSCOCO—with 16 pairs generated or selected for each strategy.The pairs serve as examples in the LLaMA in-context learning framework.Source: §C.1 · p. 16

D.1 Augmentation Qualitative Comparison

• Qualitative comparisons show that EDA and back translation mainly modify words, whereas LLM augmentation changes sentence structure while preserving key concepts and meaning.Source: §D.1 · p. 22

Augmentation strategies

• Four augmented texts per original sentence were generated for EDA, back translation, and LLaMA-based augmentation under a fair comparison.EDA applies predefined operations, back translation uses four languages, and LLaMA ICL uses four meta-input-output pairs.Source: §D.2 · p. 24
• LLM-based augmentation consistently achieves superior results across datasets and evaluation metrics, outperforming EDA and back translation.The baseline strategies improve vanilla CLIP, but the LLM-based strategy performs better overall.Source: §D.2 · p. 24

Scaling with augmentation count

• Baseline augmentation strategies do not scale well as the number of augmentations per sentence increases, indicating limited diversity in their rewritten texts.The comparison evaluates zero to four augmentations per original text on CC12M.Source: E. Number of Augmentations per… · p. 26
• LaCLIP scales well with more augmentations because LLaMA ICL produces rich and diverse rewrites, enabling continued performance improvement.This contrasts with the limited scaling observed for the baseline strategies.Source: E. Number of Augmentations per… · pp. 26–27

Feature visualization

• LaCLIP produces clearer class boundaries and more distinct clusters than other approaches across CIFAR-10, Food101, STL-10, and EuroSAT.The t-SNE visualizations suggest improved image embedding spaces for downstream tasks.Source: Figure A1 · p. 26, F. t-SNE Visualizations · p. 27

Evaluation

• The LaCLIP, LaCLIP-MT, and CLIP comparison includes zero-shot and linear-probing experiments on CC12M and RedCaps.The experiment is summarized in Table A9.Source: G. Detailed Experiment Results for… · pp. 27–28

Multi-text training

• LaCLIP-MT pairs each image with all corresponding texts, allowing the vision encoder to receive more diverse supervision during training iterations.The comparison covers pre-training on CC12M and RedCaps.Source: G. Detailed Experiment Results for… · pp. 27–28

Results

• On RedCaps, LaCLIP-MT achieves an additional 1.9% increase in zero-shot classification accuracy on ImageNet.The reported comparison is among CLIP, LaCLIP, and LaCLIP-MT.Source: G. Detailed Experiment Results for… · p. 28

Ablation setup

• The ablation studies vary LLaMA model size and temperature using ChatGPT meta-input-output prompting pairs on CC12M.The experiments use a single text augmentation strategy.Source: §I · pp. 28–29

Model size

• A 7B LLaMA model is sufficient to significantly boost LaCLIP performance, while larger models provide only modest overall benefits in these experiments.Larger models improve some downstream datasets, but their aggregate impact is relatively modest.Source: §I · p. 29

Temperature

• Higher sampling temperatures increase text diversity, but excessively high values can introduce random words or non-English tokens that harm results.Temperature controls the balance between diversity and precision during LLaMA generation.Source: §I · p. 29
• Performance is robust across temperatures from 0.3 to 1.1 and generally peaks around τ = 0.9.The authors consistently use τ = 0.9 for text rewriting in their experiments.Source: §I · p. 30, Table A12 · p. 30

Non-contrastive training

• Language augmentation improves La-Virtex over standard Virtex on PASCAL VOC07 linear classification after CC12M pre-training.This result suggests language rewrites may benefit non-contrastive vision-language training methods.Source: J. Ablation on Non-contrastive Training · p. 30, Table A13 · p. 30

Training dynamics

• LaCLIP achieves higher validation accuracy and higher training loss than CLIP across CC3M, CC12M, RedCaps, and LAION-400M.The authors interpret this pattern as improved generalization rather than improved optimization.Source: K. Training and Validation Curves · p. 31, Figure A2 · p. 31

Qualitative analysis

• LaCLIP distinguishes some fine-grained ImageNet categories where vanilla CLIP faces challenges.Figure A3 shows examples from the three categories with the largest accuracy improvements.Source: L. Visualization of Examples being… · p. 31, Figure A3 · p. 31

Text-encoder setups

• LaCLIP outperforms CLIP models using the evaluated pre-trained BERT text-encoder setups, while freezing the text encoder reduces vanilla CLIP performance.The comparison is conducted on CC12M with fine-tuned or frozen BERT-Base text encoders.Source: M. Detailed Experiment Results for… · pp. 31–32, Table A14 · p. 32