VeCLIP: Improving CLIP Training via Visual-enriched Captions

1. Introduction

• Web AltTexts may be noisy, irrelevant, or incomplete, and simple LLM rewriting cannot introduce new image-specific details; curated caption datasets are costly to scale.Prior rewriting evaluations on curated CC3M and CC12M leave suitability for extensive web-crawled data uncertain.Source: §1 · pp. 1–2
• LLM rewriting may reduce text diversity through uniform phrasing, while prior methods leave texts unaugmented and have uncertain suitability for uncensored web data.The paper motivates scalable improvements to data quality, diversity, and training methodology.Source: §1 · p. 2
• VeCap uses image-derived visual concepts and an LLM to fuse them with AltText, while VeCLIP alternates VeCap and original AltText during training.The approach retains image-text pairs rather than filtering noisy captions.Source: §1 · p. 2, Figure 1 · p. 2
• Up to 25.2% improvement over CLIP in retrieval tasks; VeCap scales beyond 300M, 5% data yields competitive retrieval, and VeCap+DFN reaches 83.1% ImageNet zero-shot accuracy@1 with H/14.The authors also report complementarity between VeCap and curated datasets.Source: §1 · p. 3

2. Related Work

• Many CLIP successors add training inputs or losses, which may adversely affect training efficiency and memory usage.Examples include self-supervised objectives, an additional decoder, or freezing the image encoder while fine-tuning text encoders.Source: §2 · p. 3
• Dataset filtering can discard image-text pairs, while LLM rewriting has shown marginal advantage on noisy web captions despite evaluations on curated CC3M and CC12M.LaCLIP [12] is cited as an LLM rewriting approach; curated datasets begin with higher-quality captions.Source: §2 · p. 3

3.1. Preliminary

• CLIP trains on batches of image-text pairs with a symmetric contrastive loss, treating matched pairs as positives and the remaining pairs as negatives.Image augmentations are applied before the vision encoder, and the image and text features are normalized.Source: §3.1 · p. 3

3.2. Recaptioning with Visual Concept Exploitation

• LLaVA generates visual concepts directly from images using prompts independent of AltText, producing captions that supply image-derived information.The pipeline then uses an LLM to check ethics and fuse concepts from AltText and visual-enriched captions into VeCap.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Scalable LLM Rewrite for Concept Fusion

• An LLM fuses AltText with LLaVA's visual concepts, retaining AltText information while restructuring the caption and helping mitigate hallucinated entities.The authors select an open-source LLM to make large-scale rewriting practical and use batch inference, reporting up to 64 times faster processing on an Nvidia A100.Source: §3.3 · pp. 4–5

3.4. VeCLIP: Mixed Training Scheme with Visual-enriched Captions for CLIP

• VeCLIP alternates AltTexts and VeCap to improve caption quality and diversity without filtering data or adding computational complexity or parameter overhead.The pipeline addresses ethical refusal and length constraints encountered during large-scale rewriting.Source: §3.4 · p. 5, §3.3 · p. 5

4.2. Results on Retrieval Tasks

• Recall@1 gains reach +16.84%/+23.26% on COCO image-to-text and +28.40%/+29.20% on Flickr30k at 3M/12M, with consistent improvements across Recall@k and both retrieval directions.The reported comparisons are against CLIP; Table 1 summarizes COCO and Flickr30k retrieval results.Source: §4.2 · pp. 5–6, Table 1 · p. 6
• GPR1200 image-to-image retrieval gains are 5.22%/3.92% at 3M/12M and average 1.84% across six domains at 200M.The authors report that the main boost comes from the Sketch domain.Source: §4.2 · p. 6, Table 2 · p. 7

4.3. Results on Image Classification

• Top-1 gains reach +10.52%/+6.42% on ImageNet/ImageNetV2 at 3M and remain +0.90%/+0.83% at 200M.Table 3 reports zero-shot Top-k accuracy on ImageNet and ImageNetV2.Source: §4.3 · p. 7, Table 3 · p. 7
• VTAB average accuracy gains exceed 5% at 3M, 12M, and 100M, and are +1.91% at 200M.The evaluation covers nine natural and specialized tasks.Source: §4.3 · pp. 7–8, Table 4 · p. 7

4.4. Performance trend across scales

• Both CLIP and VeCLIP improve strongly through 100M, after which gains become gradual and marginal; VeCLIP starts higher at 3M and 12M.Figure 4 compares CLIP with original AltTexts and VeCLIP with LLM-VeC using ViT-B/16.Source: §4.4 · p. 8, Figure 4 · p. 9

4.5. Complementary to other datasets to achieve state-of-the-art performance

• Under the same learning-rate and maximum-step configuration, VeCap beats DFN in retrieval but trails in classification; combining them gives the most improvements for all model sizes, and ViT-H/14 reaches 83.1% ImageNet accuracy.Source: §4.5 · p. 8

4.6. Ablation Study

• VeCap improves retrieval but can lag original AltTexts on ImageNet zero-shot; alternating AltTexts and VeCap raises zero-shot results while retaining retrieval gains, and using all captions performs best in nearly all settings.The authors propose that prompt distribution shift and reduced text diversity may contribute to the classification gap.Source: §4.6 · p. 8, Table 6 · p. 10
• ViT-L/14 improves up to 5.87% over ViT-B/16, while VeCap on CC3M yields +18.16% on COCO image-to-text retrieval and +6.14% on ImageNet.The CC3M results are presented as evidence of VeCap's generalizability to well-curated datasets.Source: §4.6 · pp. 8, 10

5. Discussion

• VeCLIP is designed for scalable web-crawled datasets, and the authors report performance gains while reducing the computational cost and training-data size needed for large models to reach competitive results.The evaluation spans raw and noisy datasets at small, medium, and large scales.Source: §5 · p. 10
• The authors plan to apply the collected dataset to other VLM pre-training and investigate filtering for factual inaccuracies and hallucinations in LLM outputs.The paper uses CLIP as an illustrative instance.Source: §5 · p. 10
• The study uses LLaVA to exploit visual concepts, while quality measurement for generative AI remains under study.Source: §5 · p. 10

1. Appendix

• The supplementary material provides additional dataset, experimental-setting, results, and analysis details.Source: §1 · p. 13

Pre-training datasets

• WIT-300M contains 300M web-crawled image-text pairs, with WIT-200M, WIT-100M, WIT-12M, and WIT-3M formed as successively smaller nested subsets.Source: A. Dataset details · p. 13

VTAB datasets

• VTAB evaluation uses nine zero-shot classification datasets and the same CLIP prompt set for original and proposed models.Class embeddings average the prompt-template embeddings and are then L2-normalized.Source: A. Dataset details · p. 13

Pre-training hyper-parameters

• CLIP models are pre-trained on up to 512 TPUs with JAX, with hyperparameters summarized in Table A2.Source: B. Implementation details · p. 13

C. More experimental results

• The section covers additional results and ablations, including large-backbone generalization and pre-training with public and well-curated datasets.Source: C. More experimental results · p. 13

• VeCLIP with ViT-L/14 consistently improves over CLIP with ViT-L/14 across all downstream tasks.Source: §C.1 · p. 13
• VeCLIP with ViT-L/14 surpasses its ViT-B/16 counterpart, with over 5% improvement on both ImageNet and ImageNetV2.Source: §C.1 · p. 13
• The authors report potential for VeCLIP to scale with larger backbone architectures and larger-scale datasets.Source: §C.1 · p. 13

C.2. Generalization on well-curated datasets: CC3M and CC12M

• CC3M exceeds WIT-3M by +10.70% on COCO I2T with CLIP, while VeCLIP improves Flickr30K I2T by over 30% and ImageNet and ImageNetV2 by over 5%.Source: §C.2 · p. 13
• +12.27% on COCO I2T for CC12M, with VeCLIP also improving ImageNet and ImageNetV2 by over 5%.Source: §C.2 · p. 13

C.3. Complete visual descriptions vs simplified entity representations

• Sole VeCap training may reduce zero-shot performance relative to original AltTexts, whereas the mixed training approach yields optimal outcomes.Source: §C.3 · p. 13
• Simplified Entity Representation extracts and filters noun entities into “a photo of [U],” but its zero-shot performance remains below original AltTexts; the authors hypothesize insufficient data diversity.Source: §C.3 · pp. 13–14

• WIT-300M results are close to those at 200M for CLIP and VeCLIP, suggesting 200 million samples suffice for ViT-B/16 training.Source: §C.4 · p. 14
• VeCLIP achieves significant retrieval-task improvement under the 300M setting, while its ImageNet and ImageNetV2 improvement is marginal.Source: §C.4 · p. 14
• 83.1% ImageNet accuracy is achieved by VeCLIP with DFN using ViT-H/14; the model also outperforms FLIP and OpenAI CLIP across different backbones.Source: §C.4 · p. 14

D. Caption quality comparison between well-curated Datasets and WIT datasets

• The section presents caption examples illustrating CC3M’s caption quality and VeCap’s additions, including descriptions of scenes and visible objects.Source: D. Caption quality comparison between… · p. 17
• CC3M captions are described as precise and closely aligned with images, while WIT-3M AltTexts tend to be more cluttered; VeCap adds visual concepts to both.Source: D. Caption quality comparison between… · p. 17
• The authors attribute significant retrieval-task improvement to integrating enriched visual concepts into captions.Source: Table A4 · p. 15, D. Caption quality comparison between… · p. 17

E. More examples of WIT with VeCap

• The pipeline was applied to 200 million image-text pairs, with additional WIT AltText and VeCap examples shown in Figure A1.Source: E. More examples of WIT with VeCap · p. 17, Figure A1 · p. 18