PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Multimodal Datasets with Image Captioning

Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, Ludwig Schmidt

arXiv:2307.10350v2cs.LGcs.CV

TL;DR

Web image-text datasets contain noisy captions, and filtering them can reduce data diversity. This paper studies mixing raw and image-generated captions to improve multimodal training, achieving gains across tasks while exposing scale-dependent limits of synthetic text. It also finds that standard captioning benchmarks do not reliably predict caption utility for CLIP training.

  • Problem

    Web-scraped image-text pairs are noisy, while existing filtering methods can discard useful images and reduce training-data diversity.

  • Method

    The paper generates image-conditioned captions with captioning models and evaluates strategies for filtering and mixing raw and synthetic captions across DataComp scales.

  • Results

    Across 12.8M, 128M, and 1.28B scales, synthetic-caption strategies improve multimodal training, with especially significant retrieval gains across scales.

  • Takeaways & Limitations

    Synthetic captions can restore useful supervision while combining raw and generated text helps balance image-text alignment with caption diversity.

  • Takeaways & Limitations

    At larger data quantities, the diversity gap between generated and web-scraped text and the importance of image curation constrain further gains.

Abstract

from arXiv · show

Massive web datasets play a key role in the success of large vision-language models like CLIP and Flamingo. However, the raw web data is noisy, and existing filtering methods to reduce noise often come at the expense of data diversity. Our work focuses on caption quality as one major source of noise, and studies how generated captions can increase the utility of web-scraped datapoints with nondescript text. Through exploring different mixing strategies for raw and generated captions, we outperform the best filtering method proposed by the DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a candidate pool of 128M image-text pairs. Our best approach is also 2x better at Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an effective source of text supervision. In experimenting with different image captioning models, we also demonstrate that the performance of a model on standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable indicator of the utility of the captions it generates for multimodal training. Finally, our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text, as well as the importance of image curation with increasing training data quantity. The synthetic captions used in our experiments are now available on HuggingFace.

1 Introduction

Web-scraped image-text data is noisy, while filtering can discard useful images and reduce diversity. This work uses synthetic image captions, alone and mixed with raw captions, to improve multimodal training across data scales.

  • Existing filtering reduces noise but can discard useful images and lower the diversity or concept coverage of training data.The paper frames synthetic captions as a way to restore utility to discarded examples without relying on subjective dataset construction choices.
  • At 128M samples, training on the entire candidate pool with synthetic captions outperforms raw-data filtering baselines.The comparison uses common filtering approaches such as selecting the top 30% of pairs by CLIP image-text cosine similarity.
  • Captioning-model quality for multimodal training is not reliably predicted by standard image-captioning benchmark performance.Fine-tuning toward image captioning and optimizing CIDEr can produce captions that are less effective for CLIP training, while reference-free metrics such as CLIP-S are more reliable.
  • The paper evaluates raw and synthetic-caption mixing strategies at 12.8M, 128M, and 1.28B pool sizes, with retrieval gains significant across scales.ImageNet benefits diminish as scale increases, unlike retrieval improvements.
  • Synthetic captions are individually less noisy and more visually informative, but less diverse at the population level than raw captions.Combining both sources improves caption quality in terms of text diversity and image-text alignment.
  • At larger data scales, image-based filtering and the diversity gap between generated and web-scraped captions become increasingly important.The paper identifies these factors as limitations of synthetic-caption approaches in large data regimes.

2 Related work

Prior work improves multimodal datasets through synthetic data, image captioning, and filtering. This paper differs by generating image-conditioned captions for uncurated web data rather than relying on heavily preprocessed datasets or raw-text rewriting.

  • LAION-COCO uses BLIP-generated captions on a heavily filtered 600M-pair dataset, whereas this work studies synthetic captions on uncurated web data.The paper also reports that LAION-COCO captions lag behind corresponding web-crawled captions for CLIP performance.
  • Earlier CLIP work used procedurally generated captions for 120K MS-COCO images, a scale and supervision setting unlike web-scale datasets.The paper contrasts verified multi-object labels with the much larger LAION-5B and CommonPool datasets.
  • BLIP2, Flamingo, and CoCa represent recent progress in image captioning, with synthetic captions also used in BLIP and BLIP2 training.The paper positions its caption-generation experiments within this broader captioning literature.
  • Related dataset-improvement methods filter image-text pairs using content heuristics, image properties, or learned alignment signals.Examples include CAT and filtering text regions, while this work focuses on adding image-conditioned synthetic captions to uncurated data.
  • Unlike language-model caption rewriting, image captioning generates relevant captions even when the original web text is nondescriptive.The rewriting approach assumes access to raw text and is not conditioned on the image.

3 Experiment setup

The experiments use minimally processed CommonPool data across three scales, generate captions with several models, and train fixed CLIP configurations to isolate data quality. Evaluation covers DataComp classification and retrieval tasks, using standardized cosine-similarity filtering terminology.

  • Data: The study uses CommonPool candidate pools of 12.8M, 128M, and 1.28B image-text pairs with minimal preprocessing.Preprocessing includes NSFW filtering, evaluation-set deduplication, and face blurring.
  • Captioning models: Synthetic captions are generated with BLIP, BLIP2, and OpenCLIP-CoCa, using both pretrained and fine-tuned variants.The models were pretrained on large web image-text datasets, including data containing synthetic captions.
  • Training: CLIP is trained on generated image-text datasets with fixed architectures, compute budgets, and hyperparameters at each scale.The setup trains on N samples total for a candidate pool of N pairs.
  • Evaluation: Evaluation reports ImageNet accuracy, average accuracy across 38 DataComp tasks, and Flickr30K and MS-COCO retrieval.Retrieval is the average of text-to-image and image-to-text Recall@1.
  • Terminology: “CLIP score filtering” selects examples by image-text cosine similarity, while “BLIP2” denotes captions generated with top-K sampling at temperature 0.75.These definitions apply throughout the subsequent experiments.

4 Impact of model specialization on captions generated for multimodal training

The paper tests whether captioning-model specialization and benchmark performance predict usefulness for CLIP training. Fine-tuning improves retrieval but reduces ImageNet performance, while reference-free alignment metrics better track classification utility.

  • Captioning-model specialization: Fine-tuning captioning models improves CLIP retrieval but hurts ImageNet classification, likely because generated-text diversity decreases.The diversity change is evidenced by fewer unique trigrams across 1M captions.
  • Captioning-model specialization: CIDEr does not reliably predict whether generated captions provide effective text supervision for multimodal pre-training.Models without fine-tuning have poor CIDEr despite supporting strong downstream CLIP performance.
  • Captioning metrics: Reference-based metrics such as CIDEr, SPICE, and BLEU-4 compare generated captions with human references, whereas CLIP-S evaluates image-caption compatibility.The study finds CLIP-S more reflective of ImageNet performance trends.
  • Captioning-model selection: BLIP2 without fine-tuning is selected for later raw-and-synthetic caption experiments because it performs well on both ImageNet classification and Flickr retrieval.This choice follows the comparison of pretrained and MS-COCO-fine-tuned captioning models.

5 Filtering raw and synthetic captions

The filtering study compares raw-only, filtered, and mixed caption strategies at DataComp’s medium scale. Mixing raw captions for high-scoring examples with filtered BLIP2 captions for otherwise discarded images gives the strongest reported performance, while synthetic captions are individually longer and more visually grounded.

  • Filtering strategies: The mixed strategy keeps top-x% raw-caption examples and adds BLIP2 captions for remaining images, optionally applying the same threshold to those new pairs.The compared alternatives include no filtering, CLIP-score filtering, and CLIP-score filtering intersected with ImageNet1k clustering.
  • Results: At 128M scale, mixing raw and synthetic captions with cosine-similarity filtering achieves the best ImageNet and average accuracy.Including BLIP2 captions also improves retrieval performance by more than 2× over the raw-data baseline.
  • Threshold analysis: Adding BLIP2 captions to some or all remaining images helps across filtering thresholds, but including too many raw captions can underperform using only synthetic captions.Filtering remains necessary even for synthetic captions.
  • Caption information: Synthetic captions generally contain more words and a higher fraction of visual tokens per sample than web-crawled captions.These measurements use 1M random samples and the grounding ratio for visual concepts.

6 What makes synthetic captions effective?

Synthetic captions provide better image-text alignment and can preserve more useful diversity than raw-caption filtering, improving multimodal training and especially retrieval.

  • 6.2 Performance analysis: 2× retrieval improvement is observed at DataComp’s medium scale when synthetic captions are included in training.The result averages text-to-image and image-to-text retrieval across MSCOCO and Flickr.
  • 6.1 Defining caption quality: Generated captions have higher image-text alignment than raw captions, with mean cosine similarity of 0.251 versus 0.208.The comparison uses the same sampled images and BLIP2 captions.
  • 6.1 Defining caption quality: CLIP filtering reduces caption diversity, while adding thresholded synthetic captions lowers noise and restores diversity through additional useful training data.Diversity is measured using unique trigrams, and the combined strategy respects the existing cosine-similarity threshold.
  • 6.2 Performance analysis: With similar ImageNet accuracy, BLIP2 captions improve average performance by 2.8% across 38 tasks and win on 23 tasks.The gains are especially visible on distribution shifts, text recognition, and retrieval.
  • 6.2 Performance analysis: Replacing raw captions with BLIP2 captions on the same images increases Flickr and MS-COCO retrieval performance by more than 1.5×.This isolates caption choice from image-set differences.

7 Performance at scale

Synthetic captions remain especially beneficial for retrieval as candidate pools grow, but larger-scale ImageNet performance increasingly depends on image curation and caption diversity.

  • 7 Performance at scale: At 400M and 1.28B scales, mixing curated raw and synthetic captions beats the best raw-data baseline on ImageNet by 2.5% and 1.2%, respectively.The mixed method intersects high-CLIP-score examples with ImageNet-near clusters and filters the remaining synthetic-caption examples.
  • 7 Performance at scale: Synthetic captions substantially outperform raw-text filtering baselines for retrieval at larger data quantities.The scale analysis focuses on ImageNet classification and Flickr retrieval; MS-COCO retrieval is excluded from this comparison because it was in BLIP2 pre-training data.
  • 7 Performance at scale: At larger scales, the diversity gap between generated and web-scraped text and the increasing importance of image quality limit gains from text supervision alone.These effects motivate improving generated-caption diversity and curating image content.
  • 7 Performance at scale: Across scales, replacing noisy raw captions with filtered synthetic captions improves classification and retrieval under a fixed training budget.The authors identify this mixed strategy as their best medium-scale baseline.
  • 7 Performance at scale: The best ImageNet filtering approach changes with candidate-pool scale, whereas the best Flickr retrieval method consistently uses synthetic captions.Image-based curation becomes increasingly important for ImageNet as scale grows.

8 Conclusion

The paper concludes that synthetic captions improve multimodal training and retrieval, especially at small and medium scales, while larger datasets expose limits from reduced text diversity and image quality.

  • 8 Conclusion: Synthetic captions improve caption quality for multimodal training and enhance capabilities such as retrieval.The conclusion frames these as the main demonstrated benefits.
  • 8 Conclusion: Generated captions are highly effective at small and medium scales, but their diversity gap with web-scraped text hinders gains at larger quantities.At large scale, improving text supervision alone also makes state-of-the-art ImageNet accuracy harder to obtain.
  • 8 Conclusion: The experiments do not cover an exhaustive set of available image-captioning systems.The paper instead focuses on understanding how to combine raw and synthetic caption signals.
  • 8 Conclusion: Generated captions can inherit captioning-model biases, and training subsequent models on them may amplify those biases.The paper also notes that harmful stereotypes may remain in multimodal web datasets.
  • 8 Conclusion: Future work includes increasing generated-caption diversity through sampling changes or multiple captioning systems.The authors also propose better algorithms for combining raw and generated captions.

A More examples of image-text pairs (no cherry picking)

The examples compare raw web text with captions generated by several models, showing that synthetic captions often describe visible content more directly than noisy or irrelevant source text.

  • A More examples of image-text pairs (no cherry picking): Synthetic captions frequently replace opaque product, location, or metadata text with descriptions of visible objects, people, and scenes.Examples include cars, buildings, watches, food, clothing, and people in indoor or outdoor settings.
  • A More examples of image-text pairs (no cherry picking): Fine-tuned captions generally describe image content but can still be generic, mistaken, or less informative than alternative generated captions.Examples include misidentifying scenes, objects, or activities and producing broad descriptions such as a group of people or a large building.
  • A More examples of image-text pairs (no cherry picking): Captioning models differ in specificity: some provide broad scene descriptions, while others add product names, activities, or contextual details.The examples show variation across BLIP, BLIP2, and OpenCLIP-CoCa, including their fine-tuned versions.
  • A More examples of image-text pairs (no cherry picking): Some generated captions preserve text visible in images, such as labels, buttons, names, pronouns, and loading screens.These examples illustrate caption behavior on images dominated by embedded text or interface elements.

B Experiment details

Experiments use fixed DataComp configurations across three scales, with substantial training and annotation costs. The added annotation cost can be amortized through repeated training or reuse of the improved dataset.

  • Training configuration: Small, medium, and large training use ViT-B/32, ViT-B/32, and ViT-B/16 respectively, with batch sizes of 4096, 4096, and 8192.The shared optimizer is AdamW with β2 = 0.98, learning rate 5e-4, and 500 warmup steps.
  • Compute cost: 4, 40, and 960 A100 hours are required for small, medium, and large DataComp training runs.
  • Annotation cost: BLIP2 caption and CLIP score generation costs 6× a single DataComp training run.The authors argue this cost can be amortized across additional epochs and downstream models trained on the improved dataset.
  • Cost trade-off: Synthetic-caption annotation costs can be amortized by training for more epochs or reusing the improved dataset for multiple downstream models.

C Temperature ablations

Temperature and data-mixing analyses compare captioning and filtering choices across ImageNet, retrieval, and broader DataComp evaluations. The experiments also examine retrieval gains, class-level behavior, and caption-diversity scaling.

  • Temperature ablations: T = 0.75 and T = 1.0 generally lead to good CLIP-training performance across ImageNet and averaged 38-task accuracy.
  • Retrieval: Mixing BLIP2 captions with raw captions significantly boosts Flickr and MS-COCO retrieval performance.
  • Caption mixing: Adding BLIP2 captions for discarded images improves performance on 30 of 38 tasks and raises average accuracy by 4%.The comparison uses the top 30% of raw-caption examples versus that set augmented with BLIP2 captions for the remaining images.
  • Class-level analysis: Models trained on raw versus synthetic captions show no systematic ImageNet-classification trend for living versus non-living categories.
  • Caption diversity: Synthetic captions scale worse than raw captions in unique nouns and unique trigrams, widening the data-diversity gap as dataset size increases.The authors suggest this increasing gap may affect synthetic-caption performance benefits.

G Experiments with LAION-COCO

LAION-COCO experiments test whether newer captioning models can make synthetic captions competitive with curated raw captions. BLIP2 closes most of the gap, especially after controlling caption noise, while the benchmark’s heavy curation limits its suitability for studying synthetic-caption utility.

  • Captioning-model comparison: BLIP2 closes 75% of the ImageNet performance gap with BLIP captions and approaches raw-caption average accuracy on LAION-COCO.The BLIP2 captions were generated using one configuration because of compute constraints.
  • Noise-controlled comparison: After applying the same CLIP-score threshold, BLIP2 is about as effective as raw captions, with -0.7% ImageNet accuracy and +0.2% average accuracy.This comparison uses the resulting 41M images to control for caption noise quality.
  • Benchmark limitation: LAION-COCO is unsuitable for studying synthetic-caption utility because its raw captions are already heavily filtered and discarded datapoints are unavailable.Approximately 90% of Common Crawl data is discarded before the retained LAION data is used.
  • Interpretation: The experiments support progress in the BLIP model family as a route to generated text supervision that rivals raw captions for CLIP training.
  • Fairness analysis: Fairface evaluations report improved minority-group classification and reduced male–female performance gaps with synthetic captions, while noting limitations in the category definitions.The authors leave broader fairness implications for future work.
Loading 2307.10350v2…