Source-linked AI summary
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, Arun Sacheti
TL;DR
Vision-language pre-training needs stronger joint representations and larger image-text datasets than commonly available. The paper introduces Transformer-based ImageBERT, trains it with four cross-modal objectives and a two-stage LAIT-to-CC/SBU strategy, and reports state-of-the-art retrieval results on MSCOCO and Flickr30k.
Problem
Cross-modal pre-training requires large-volume, high-quality image-text pairs, while commonly used datasets remain limited for increasingly large models.
Method
ImageBERT is a Transformer-based joint-embedding model pre-trained with MLM, MOC, MRFR, and ITM, using LAIT first and Conceptual Captions plus SBU Captions second.
Results
ImageBERT achieves new state-of-the-art results on image retrieval and sentence retrieval on MSCOCO and Flickr30k.
Takeaways & Limitations
Large-scale weakly supervised out-of-domain data can add value to pre-trained model quality and benefit downstream vision-language tasks.
Abstract
from arXiv · showhide
In this paper, we introduce a new vision-language pre-trained model -- ImageBERT -- for image-text joint embedding. Our model is a Transformer-based model, which takes different modalities as input and models the relationship between them. The model is pre-trained on four tasks simultaneously: Masked Language Modeling (MLM), Masked Object Classification (MOC), Masked Region Feature Regression (MRFR), and Image Text Matching (ITM). To further enhance the pre-training quality, we have collected a Large-scale weAk-supervised Image-Text (LAIT) dataset from Web. We first pre-train the model on this dataset, then conduct a second stage pre-training on Conceptual Captions and SBU Captions. Our experiments show that multi-stage pre-training strategy outperforms single-stage pre-training. We also fine-tune and evaluate our pre-trained ImageBERT model on image retrieval and text retrieval tasks, and achieve new state-of-the-art results on both MSCOCO and Flickr30k datasets.
1 Introduction
Vision-language tasks span retrieval, question answering, reasoning, and captioning, motivating models that learn joint image-text representations rather than relying only on late fusion. ImageBERT is proposed as a Transformer-based cross-modal pre-trained model and achieves state-of-the-art retrieval results on MSCOCO and Flickr30k.
- Vision-language tasks include text-image retrieval, VQA, VCR, and image captioning.
- Earlier methods commonly built on separately pre-trained language and vision models and used late fusion.
- ImageBERT learns joint vision-language representations through cross-modal pre-training with a Transformer architecture.
- ImageBERT achieves new state-of-the-art results on text-to-image and image-to-text retrieval tasks on MSCOCO and Flickr30k.
2 Related Work
Related cross-modal pre-training work differs in how it combines modalities, represents image content, and selects pre-training data. The literature emphasizes region-based visual tokens and the importance of data quality and volume.
- Model architecture: Cross-modal models use either separate modality-specific Transformers followed by cross-modal fusion or a single Transformer over concatenated image and sentence inputs.
- Image visual tokens: Most recent systems represent images with detected regions of interest that serve as visual tokens alongside linguistic tokens.
- Pre-train data: Conceptual Captions contains 3M image descriptions, while UNITER combines four datasets into a 9.6M training corpus.
- Pre-train data: The related work argues that both data quality and data volume are important for vision-language model training.
3 Large-Scale Weak-supervised Image-Text Data Collection
The paper constructs LAIT by weakly supervising large-scale web image-text collection through webpage, image, sentence, semantic, and aggregation filters. LAIT contains 10M image-text pairs and is intended to improve vision-language pre-training.
- Existing cross-modal pre-training datasets are smaller than desired for models with several hundred million parameters or more.
- LAIT contains 10M images with descriptions averaging 13 words, collected through a weakly supervised web-data approach.
- The pipeline crawls billions of webpages, keeps English content, extracts image URLs, and detects dominant images using HTML and DOM features.
- Image filtering retains images larger than 300 pixels per dimension and removes racy, pornographic, unnatural, non-realistic, and non-learnable images.
- Sentence cleaning uses HTML metadata and surrounding text, heuristic noise filtering, length constraints, and an out-of-vocabulary threshold.
- A weak image-text semantic model filters irrelevant pairs using text-only, image-content, and cross-modal features.
- When images have multiple descriptions, the pipeline keeps the highest-scored pair and drops pairs when too many images share one description.
4 ImageBERT Model
ImageBERT uses a Transformer to jointly model visual region and linguistic tokens, with four pre-training tasks for cross-modal representation learning. It is trained through multiple stages and then fine-tuned for image-text retrieval.
- 4.1 Embedding Modeling: ImageBERT feeds Faster-RCNN region features and textual tokens into a bidirectional self-attention Transformer to model cross-modal relationships.Both modalities are projected into compatible embeddings before joint processing.
- 4.1 Embedding Modeling: Visual tokens use region features and spatial information, while linguistic tokens combine WordPiece word, segment, and position embeddings.Detected-region order is not assumed; visual tokens use a fixed dummy position and encoded coordinates.
- 4.2 Multi-stage Pre-training: ImageBERT first pre-trains on LAIT, then on Conceptual Captions and SBU Captions; this multi-stage strategy performs better than single-stage training.The final retrieval fine-tuning stage removes masked-word and masked-object tasks and uses image-text retrieval objectives.
- 4.3 Pre-training tasks: The model uses four pre-training tasks: MLM, MOC, MRFR, and ITM, covering language, visual content, and image-text alignment.MLM predicts randomly masked words, MOC predicts masked object categories, MRFR regresses masked region features, and ITM classifies matched versus mismatched pairs.
- 4.3 Pre-training tasks: MLM masks 15% of input sub-word tokens, while MOC masks object tokens and predicts their Faster-RCNN classification categories.MOC masks each object token with probability 15% and uses cross-entropy over object classes.
- 4.3 Pre-training tasks: MRFR regresses the embedding feature of each masked object with an L2 loss, and the three masked objectives use conditional masking only for related image-text inputs.A fully connected layer projects the predicted feature to the pooled input RoI feature dimension.
5 Experiments
ImageBERT is evaluated through zero-shot and fine-tuned image-text retrieval on Flickr30k and MSCOCO, with ablations examining pre-training data, losses, visual features, region counts, and fine-tuning losses. The results support multi-stage pre-training and show state-of-the-art fine-tuned retrieval performance on both datasets.
- Evaluation settings: Evaluation covers zero-shot and fine-tuned image-text retrieval on Flickr30k and MSCOCO, including image retrieval and text retrieval.Zero-shot evaluation measures pre-training quality, while fine-tuning evaluates downstream retrieval performance.
- Fine-tuned results: ImageBERT achieves new state-of-the-art results on both Flickr30k and MSCOCO across 1k and 5k test sets after fine-tuning.The model outperforms the other methods reported in the comparison.
- Pre-training data ablation: Multi-stage LAIT → CC+SBU pre-training performs significantly better than the other tested dataset combinations.The comparison includes single-dataset training and a single-stage combination of LAIT, Conceptual Captions, and SBU Captions.
- Visual representation ablations: Global image features do not improve all metrics, whereas increasing the number of detected RoIs can improve results by providing more image context.The model uses 100 top-ranked RoIs in the main experiments, and the global-feature study tests DenseNet, ResNet, and GoogleNet features.
- Pre-training loss ablation: Adding MRFR loss produces a huge improvement in zero-shot results, indicating that the harder visual-content modeling task contributes to joint learning.The ablation compares pre-training configurations with and without MRFR loss.
- Fine-tuning loss ablation: Binary cross-entropy loss alone gives the best fine-tuned image-text retrieval results among the tested fine-tuning loss combinations.This comparison is reported in the fine-tuning loss ablation.
6 Conclusion
ImageBERT combines Transformer-based vision-language joint embedding with a large-scale weakly supervised LAIT corpus in a multi-stage pre-training pipeline. It achieves state-of-the-art image and sentence retrieval results on MSCOCO and Flickr30k.
- ImageBERT is a Transformer-based model for vision-language joint embedding.
- LAIT is a large-scale weakly supervised image-text corpus collected from the Web for the pipeline’s first pre-training stage.
- Large-scale out-of-domain data with imprecise human labels can improve pre-training quality and benefit downstream tasks.
- ImageBERT achieves new state-of-the-art results on image retrieval and sentence retrieval tasks on MSCOCO and Flickr30k.