Source-linked AI summary
Scaling Up Vision-Language Pre-training for Image Captioning
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, Lijuan Wang
TL;DR
Image-captioning VLP had limited evidence about how pre-training data and model size scale together, especially beyond moderate models and datasets. The paper introduces LEMON and ALT200M to study this systematically, finding new state-of-the-art results across major benchmarks and strong long-tail concept recognition. Its conclusions are bounded by nocaps’ dependence on external resources for novel-object recognition and by comparisons that account for differing pre-training data.
Problem
The impact of pre-training dataset size and its relationship with model settings remains unclear for image captioning, despite scaling’s importance in VLP.
Method
LEMON studies VLP scaling by varying model size and pre-training data, using VinVL as a reference and ALT200M web alt-text pairs up to 200 million.
Results
LEMON achieves new state-of-the-art results on COCO Caption, nocaps, and Conceptual Captions, with large-scale pre-training improving recognition of long-tail visual concepts.
Takeaways & Limitations
Increasing model and pre-training data sizes together can improve captioning performance and avoid a saturation plateau, while large-scale noisy alt-text supervision supports zero-shot long-tail concept recognition.
Takeaways & Limitations
Nocaps evaluation emphasizes external resources because its out-domain objects are absent from COCO training data, so results depend on extra image-caption pairs.
Abstract
from arXiv · showhide
In recent years, we have witnessed significant performance boost in the image captioning task based on vision-language pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art VinVL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million image-text pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail visual concepts when used in a zero-shot manner.
1. Introduction
The paper examines how pre-training data and model scale affect vision-language pre-training for image captioning, an underexplored question despite scale’s importance. It introduces LEMON and studies scaling with up to 200 million web image-text pairs and transformer models from 13 to 675 million parameters.
- VLP has improved image captioning, but the impact of pre-training dataset size and its interaction with model settings remains unclear.
- Prior scaling studies report benefits from larger models with massive data, while captioning-specific scaling properties remain insufficiently studied.
- LEMON constructs ALT200M, a web-collected dataset containing up to 200 million image-text pairs derived from image alt attributes.
- The study scales VinVL’s transformer width and depth across model sizes ranging from 13 million to 675 million parameters and uses subsets containing 3 to 200 million pairs.
- Its proposed scaling rule combines larger models and more pre-training data to improve performance and avoid a saturation plateau.
- The paper reports new state-of-the-art image-captioning results across COCO Caption, nocaps, and Conceptual Captions.
2. Related Work
Related work has advanced vision-language pre-training and empirical scaling laws, but scaling remains less studied for image captioning than for retrieval and representation learning. This paper addresses that gap by varying both data and model sizes and comparing against prior large-scale systems.
- Vision-language pre-training has produced prominent multimodal models and a broad body of work on training strategies, robustness, compression, probing, and video-text modeling.
- Most VLP studies use base or large transformers and no more than 4 million images, leaving larger-scale captioning behavior less explored.
- CLIP and ALIGN scaled contrastive pre-training to 400 million and 1.8 billion images for retrieval-oriented image-text representation learning.
- LEMON studies image-captioning scaling by independently altering data and model sizes, unlike CLIP and ALIGN’s retrieval focus and SimVLM’s lack of data-scale analysis.
- Dataset comparisons emphasize ALT200M’s long-tail vocabulary, including the number of unigrams in the 0.1% occurrence tail.
- Scaling-law research links performance to dataset size, model size, computation, and model shape across language and vision settings.
3. Method
LEMON constructs a large-scale image-text pre-training dataset and scales a VinVL-based captioning transformer across model sizes. Its sequence-to-sequence masked language modeling objective supports autoregressive caption generation while accommodating noisy captions.
- 3.1. Pre-training Dataset: ALT200M contains more than 200 million images, each paired with one automatically collected alt-text.The collection pipeline crawls Internet images and associated alt attributes with minimal rule-based filtering.
- 3.1. Pre-training Dataset: ALT200M has nearly 16× more images than CC12M and an average caption length of 13.01 versus 10.44 for COCO captions.Its vocabulary is almost doubled, and 56% of unigrams account for only 0.1% of occurrences, indicating a long tail.
- 3.2. VLP Model for Captioning: The model fuses image regions, object tags, and caption tokens with stacked transformer encoder layers using multi-head self-attention.Faster R-CNN extracts image-region features, while WordPiece tokenizes captions and object tags.
- 3.2. VLP Model for Captioning: LEMON scales the transformer across eight configurations from 13M to 674M parameters while retaining the VinVL reference architecture.The transformer is scaled in both width and depth.
- 3.3. Training Objective: Sequence-to-sequence masked language modeling randomly masks 15% of caption tokens and trains the model to recover masked or polluted tokens causally.The same cross-entropy loss is used during pre-training and finetuning, and inference generates tokens left to right by recovering a terminal [MASK].
- 3.3. Training Objective: The study compares masked-language modeling with unidirectional language modeling and also evaluates an encoder-decoder transformer structure.The alternative objectives and structures are tested with base-sized models and swept hyperparameters.
4. Experiments
Experiments evaluate captioning under varying data scales, model sizes, training objectives, and datasets. Results show that larger-scale pre-training and greater model capacity improve benchmark performance, sample efficiency, out-of-domain transfer, and zero-shot recognition.
- Setup: nocaps evaluates long-tail visual concepts across in-domain, near-domain, and out-of-domain images, emphasizing recognition beyond COCO's annotated objects.The out-domain contains objects unseen in COCO, making external image-text resources important for evaluation.
- Setup: ALT200M pre-training uses randomly sampled web alt-text subsets from 3 to 200 million images, excluding COCO and Visual Genome from the scaling experiments.Larger datasets are supersets of smaller ones, and models are generally trained from scratch.
- Captioning Results: 16.3 CIDEr points on nocaps in-domain and 45.3 points out-of-domain separate ALT200M-pre-trained LEMON from the COCO-only baseline.The out-of-domain gain is substantially larger, supporting improved recognition of long-tailed visual objects.
- Scaling Analysis: The larger model benefits more from additional pre-training data, with the size gap becoming especially pronounced on nocaps out-of-domain images.On COCO, the small-large gap is negligible at 3M data but grows as the data scale increases.
- Scaling Analysis: At 3M pre-training data, base capacity is sufficient, whereas beyond 40M larger models significantly outperform smaller ones.The results suggest model capacity becomes a bottleneck as available data reaches hundreds of millions.
- Data Quality: ALT200M and CC12M produce similar results at 12M scale, differing by only 0.1 to 0.3 CIDEr on COCO and nocaps.The authors attribute later improvements to pre-training scale rather than a data-quality difference at this scale.
- Sample Efficiency: The larger model reaches comparable COCO results with 2 to 8 times fewer pre-training samples than the base model, with an even larger advantage on nocaps.The base model seeing 19 billion samples remains slightly below the huge model seeing 0.8 billion samples.
- Training Objectives: s2s-MLM eventually surpasses LM after COCO fine-tuning, although LM converges faster initially and is more sensitive to learning rates.On COCO, the four compared settings differ by 1.41% relative CIDEr, while CC3M shows a larger 9.10% relative change.
5. Conclusions
The paper studies VLP scaling for image captioning through ALT200M and LEMON, finding that more pre-training data and model capacity improve captioning and long-tail recognition. LEMON achieves state-of-the-art results across major benchmarks and supports zero-shot visual concept recognition.
- Conclusions: LEMON studies VLP scaling for image captioning while constructing the ALT200M large-scale dataset.The experiments vary both pre-training data and transformer capacity.
- Conclusions: Scaling pre-training produces remarkable downstream gains and state-of-the-art results on COCO Caption, nocaps, and Conceptual Captions.The conclusion reports improvements across multiple image-captioning benchmarks rather than a single evaluation setting.
- Conclusions: LEMON recognizes a wide range of long-tail visual objects even without fine-tuning.This zero-shot capability extends the model's demonstrated use beyond standard supervised captioning evaluation.
- Conclusions: With orders of magnitude more training data, transformer capacity becomes a bottleneck for exploiting alt-text data.The authors identify substantially larger models as a promising direction for further scaling.
A. Related Work on Image Captioning
Image-captioning research has explored diverse model structures and learning approaches, including attention mechanisms, semantic attributes, scene graphs, and graph convolutions. These methods generally rely on well-annotated captions from relatively small datasets, whereas this work studies much larger, noisier data.
- Attention-based models such as ORT, AoANet, M2 Transformer, X-LAN, and RSTNet improve image captioning through enhanced attention mechanisms.
- Other approaches incorporate semantic attributes, scene graphs, or graph convolutional networks for caption generation.
- This work differs by investigating data scale with much larger and noisier datasets than the small, well-annotated captioning benchmarks used by prior methods.
B. Comparison of Image-Text Datasets
Existing image-text datasets differ in scale, collection pipelines, accessibility, and language coverage. ALT200M is constructed to support scaling studies by retaining large quantities of minimally filtered web data, while its associated analyses emphasize both breadth and noise.
- WIT contains 11.5 million unique images and 37.6 million texts spanning 108 languages, collected from Wikipedia content pages.
- WenLan has 30 million web-collected image-text pairs processed with elaborate cleaning and topic-model-based content selection.
- LAION-400M uses CLIP-based filtering, whereas other datasets apply expert heuristics, complex models, or rule-based filters during collection.
- ALT200M applies minimal rule-based filters to web-crawled images and alt attributes, prioritizing retention of data for scaling experiments.
- Figure 8 compares nocaps performance across model sizes, plotting parameter counts on a logarithmic x-axis for each dataset size.
C. Detailed Hyperparameters
The training setup uses fixed pre-training and fine-tuning schedules across model sizes, with distinct learning-rate procedures for the two stages. Evaluation results are reported on COCO and nocaps.
- All models are pre-trained for 60 epochs on ALT200M subsets and fine-tuned for 40 epochs on COCO with batch size 512.
- During pre-training, the learning rate warms up for 2% of steps, reaches a peak, and then linearly decays to zero.
- During fine-tuning, the learning rate linearly decays from its initial value to zero without warm-up.
- The reported evaluations use the COCO Karpathy test split and the nocaps validation set.
D. More Qualitative Examples
The qualitative examples illustrate both the noisy supervision in ALT200M and LEMON's ability to generate detailed captions after large-scale pre-training. The examples also compare zero-shot and COCO-fine-tuned outputs.
- ALT200M includes descriptive captions, short phrases, malformed text, and text that does not precisely describe the image.
- Some ALT200M texts mention external knowledge or news content rather than directly describing visible image content, challenging learning from noisy supervision.
- Despite this noise, the dataset contains diverse fine-grained concepts such as burdock leaf, karate, mahjong, and great blue heron.
- After ALT200M training, LEMON can recognize more fine-grained objects than a COCO-only baseline, including in zero-shot use.
- Figure 10 presents additional LEMON captions generated either zero-shot or after fine-tuning on COCO.