Source-linked AI summary
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, Ming Zhou
TL;DR
Cross-modal tasks need models that jointly handle long language sequences and visual content, which separate vision and language pre-training does not provide. Unicoder-VL addresses this with a Transformer trained on image-caption pairs using MLM, MOC, and VLM, and transfers it to retrieval and VCR with strong results. The paper reports state-of-the-art or comparable performance across these tasks, while leaving image-only pre-training as future work.
Problem
Existing vision and language pre-trained models do not directly provide joint representations for cross-modal tasks with long natural-language inputs.
Method
Unicoder-VL feeds visual and linguistic content into a multi-layer Transformer and applies MLM, MOC, and VLM during pre-training on image-caption pairs.
Results
Unicoder-VL achieves state-of-the-art retrieval results and comparable visual commonsense reasoning results, including direct zero-shot retrieval performance.
Takeaways & Limitations
Cross-modal pre-training transfers to image-text retrieval and visual commonsense reasoning with simple task-specific modifications.
Takeaways & Limitations
The paper leaves extending Unicoder-VL to image-only tasks as future work and does not establish how such inputs should be used in pre-training.
Abstract
from arXiv · showhide
We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly. The last task tries to predict whether an image and a text describe each other. After pretraining on large-scale image-caption pairs, we transfer Unicoder-VL to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer. We achieve state-of-the-art or comparable results on both two tasks and show the powerful ability of the cross-modal pre-training.
Introduction
Existing vision and language pre-trained models handle their own modalities well but do not directly model long-sequence vision-language interactions. Unicoder-VL addresses this with joint cross-modal pre-training and achieves strong transfer results.
- Motivation: Vision pre-trained models trained on image labels cannot directly handle cross-modal tasks with long natural-language inputs.Such tasks therefore require additional fusion layers to model visual-linguistic interactions.
- Motivation: Language pre-trained models handle long natural-language sequences but are not trained directly with visual content.They learn contextualized text representations from large-scale corpora before downstream fine-tuning.
- Unicoder-VL: Unicoder-VL uses a multi-layer Transformer to learn joint vision-language representations through cross-modal pre-training on large-scale image-caption pairs.The framework is inspired by BERT and cross-lingual pre-trained models such as XLM and Unicoder.
- Unicoder-VL: Three tasks are used: Masked Language Modeling, Masked Object Classification, and Visual-linguistic Matching.MLM and MOC learn context-aware token representations jointly, while VLM predicts whether an image and text describe each other.
- Results: State-of-the-art results were achieved on MSCOCO and Flickr30K retrieval, with comparable results on visual commonsense reasoning.The model also showed good zero-shot performance, indicating generalization ability.
Related Work
Prior vision and language pre-training approaches differ in architecture, masking objectives, and pre-training data. Unicoder-VL uses a single-stream cross-modal design with direct visual features and attains strong retrieval and VCR results.
- Pre-training for CV Tasks: ImageNet-based vision models are poorly suited to cross-modal tasks requiring long natural-language inputs because they are trained only on image labels.Unicoder-VL instead uses image-caption pairs, making it more suitable for these tasks.
- Pre-training for NLP Tasks: NLP pre-trained models learn contextualized text representations from large-scale corpora and adapt to downstream tasks through fine-tuning.Their pre-training does not directly include visual content.
- Cross-modal Pre-training: VideoBERT combines transcript words with generated visual words, whereas Unicoder-VL jointly trains on visual object features and linguistic content.This distinguishes direct visual-feature modeling from discretized visual-word representations.
- Architectures: ViLBERT uses separate vision and language networks followed by cross-modal Transformers, while Unicoder-VL belongs to the single-stream architecture group.VisualBERT, VL-BERT, and UNITER are also described as single-stream models.
- Pre-training Tasks: The compared models vary in object prediction, masking strategy, and use of visual-linguistic matching during pre-training.All listed models use masked language modeling, but their treatment of object prediction and modality masking differs.
- Results: Unicoder-VL achieves state-of-the-art results on image-to-text retrieval, text-to-image retrieval, and VCR compared with recent works.The paper presents this as evidence of the model’s ability on these tasks.
Approach
Unicoder-VL encodes image regions and text tokens together with a self-attention Transformer, then pre-trains the joint representation using three cross-modal objectives. Visual inputs come from fixed Faster R-CNN region features.
- Model Background: BERT provides the multi-layer Transformer foundation and uses masked language modeling and next sentence prediction during pre-training.Unicoder-VL extends this foundation to visual and linguistic inputs.
- Unicoder-VL Encoder: Image regions and textual tokens are encoded and passed through multi-layer self-attention to learn cross-modality contextualized embeddings.The resulting embeddings jointly represent visual regions and textual tokens.
- Input Embeddings: Text inputs use WordPiece tokens with [CLS], [SEP], and [IMG] special tokens, while linguistic representations combine word and position embeddings.The embeddings are initialized from BERT.
- Input Embeddings: Image inputs use Faster R-CNN to extract pooled region features and location features, which are projected into a shared embedding space.The final region representation sums the projected visual and location features before layer normalization.
- Input Embeddings: The Faster R-CNN model remains fixed during Unicoder-VL training.Its predicted object labels are retained for the object-label prediction task.
- Pre-training Tasks: The three pre-training tasks are Masked Language Modeling, Masked Object Classification, and Visual-linguistic Matching.Together they provide language masking, object prediction, and image-sentence alignment objectives.
- Pre-training Tasks: MLM masks words with probability 15% and predicts them from surrounding words and all image regions.The objective is a negative log-likelihood over masked words.
- Pre-training Tasks: MOC masks sampled visual features and predicts their object categories from the Transformer output.The target category is the highest-confidence label predicted by the detection model.
Experiments
The experiments pretrain and transfer Unicoder-VL across image-text retrieval and visual commonsense reasoning, using large-scale image-caption data and task-specific fine-tuning procedures.
- Pretraining setup: 3.8M image-caption pairs are used for pretraining, combining approximately 3M Conceptual Captions pairs with 0.8M SBU pairs.The model uses 12 Transformer layers, 768 hidden units, 12 attention heads, and a maximum sequence length of 144.
- Image-Text Retrieval: Retrieval evaluation uses MSCOCO and Flickr30K, with R@1, R@5, and R@10 measuring the percentage of ground-truth matches in the top K results.MSCOCO and Flickr30K provide separate training, validation, and testing splits for evaluating both retrieval directions.
- Image-Text Retrieval: Image-text retrieval is formulated as ranking with image-to-text and text-to-image matching tasks, sampling three negative cases per matching task.Triplet loss maximizes the margin between positive and negative pairs after computing cross-modal similarity scores.
- Image-Text Retrieval: The zero-shot retrieval setting applies the pretrained visual-linguistic alignment mechanism directly without task-specific fine-tuning.This setting tests whether pretraining can ground text and generalize across visual and linguistic variation.
- Visual Commonsense Reasoning: VCR fine-tuning concatenates each question with candidate answers or rationales, passes four image-text inputs through the model, and predicts with a softmax over four scores.Ground-truth object boxes are matched to extracted visual features, and referenced visual features are additionally attached to corresponding language tokens.
Results and Analysis
Unicoder-VL’s pre-train-plus-fine-tune setting outperformed retrieval baselines across evaluation metrics, with larger gains on Flickr30k, while VCR improvements were comparatively modest. Ablations indicate benefits from larger Transformer encoders and more pre-training data, alongside limitations from domain mismatch and unfine-tuned detection backbones.
- Image-Text Retrieval: Unicoder-VL’s pre-train-plus-fine-tune setting significantly outperformed all baselines on every image-text retrieval evaluation metric.The comparison covers zero-shot, task-specific training, and pre-train-plus-fine-tune settings.
- Image-Text Retrieval: 7.8% and 8.1% absolute R@1 improvements over PFAN were obtained on the MSCOCO 1K sentence-retrieval and image-retrieval tasks, respectively.On Flickr30k, Unicoder-VL reached 86.2% and 71.5% R@1 for sentence and image retrieval, with larger gains over PFAN.
- Image-Text Retrieval: 10.1 points of improvement over ViLBERT were reported, while Unicoder-VL achieved comparable results to UNITER despite using 1.8M fewer image-caption pairs.The authors note that UNITER’s additional data included in-domain Visual Genome captions.
- Visual Commonsense Reasoning: Unicoder-VL only slightly improved VCR performance with pre-training, although it outperformed R2C by large margins using only a simple classification layer.The authors attribute the limited pre-training gain to a gap between perceptual image-caption data and cognitively oriented VCR data.
- Discussion: The method does not use image-only inputs because training failed to converge, and its Faster-RCNN backbone was not fine-tuned jointly with cross-modal training.The authors leave extension to image-only tasks and detection-backbone fine-tuning for future work.
- Ablation Studies: Larger Transformer encoders improved image-text retrieval, and retrieval accuracy increased monotonically as the pre-training dataset grew.The model-depth ablation used 6-, 12-, and 24-layer encoders; the dataset-size study used 75% of the full dataset as one comparison.
Conclusion
Unicoder-VL uses large-scale image-caption pretraining and three cross-modal tasks to learn aligned visual-linguistic representations. Fine-tuning demonstrates strong retrieval gains, zero-shot transfer, and improved visual commonsense reasoning.
- Unicoder-VL pretrains on large-scale image-caption pairs with three tasks that align visual and linguistic modalities.The tasks are introduced to learn better cross-modal representations.
- Unicoder-VL significantly boosts image and sentence retrieval performance on Flickr30K and MSCOCO after fine-tuning.
- Zero-shot experiments show that Unicoder-VL applies learned cross-modal knowledge directly to image and sentence retrieval without task-specific fine-tuning.
- Cross-modal pretraining improves visual commonsense reasoning, although the method is presented as broadly applicable beyond the evaluated tasks.