Source-linked AI summary
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu
TL;DR
Vision-language models commonly rely on region-based visual features that are task-specific and limited by detection categories. Pixel-BERT aligns image pixels with text using a CNN visual encoder, BERT token embeddings, multi-modal Transformers, and self-supervised pre-training on image-sentence pairs. The model achieves leading results across several downstream tasks, including a 2.17-point single-model VQA improvement over prior SOTA under fair comparison.
Problem
Region-based visual representations are designed for specific visual tasks and detection categories, limiting the visual semantics available for vision-language learning.
Method
Pixel-BERT uses a CNN visual encoder with BERT token embeddings and multi-modal Transformers, pre-trained on image-sentence pairs with Masked Language Modeling, Image-Text Matching, and random pixel sampling.
Results
Pixel-BERT achieves the best or state-of-the-art performance across VQA, NLVR2, and image-text retrieval, improving single-model VQA by 2.17 points over previous SOTA under fair comparison.
Takeaways & Limitations
Using source-image pixels instead of detection bounding-box features provides a universal visual-language representation for downstream vision-language tasks.
Takeaways & Limitations
The method uses random pixel sampling instead of masked visual prediction because pixel reconstruction is more difficult than region reconstruction, leaving visual self-supervised task design for future work.
Abstract
from arXiv · showhide
We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of bounding box annotations and overcomes the unbalance between semantic labels in visual task and language semantic. To provide a better representation for down-stream tasks, we pre-train a universal end-to-end model with image and sentence pairs from Visual Genome dataset and MS-COCO dataset. We propose to use a random pixel sampling mechanism to enhance the robustness of visual representation and to apply the Masked Language Model and Image-Text Matching as pre-training tasks. Extensive experiments on downstream tasks with our pre-trained model show that our approach makes the most state-of-the-arts in downstream tasks, including Visual Question Answering (VQA), image-text retrieval, Natural Language for Visual Reasoning for Real (NLVR). Particularly, we boost the performance of a single model in VQA task by 2.17 points compared with SOTA under fair comparison.
1 Introduction
Pixel-BERT addresses limitations of region-based visual representations by aligning image pixels and text with an end-to-end CNN and multi-modal Transformer framework. Pre-training with image-sentence pairs and self-supervised objectives achieves strong results across vision-language tasks, including a 2.17-point VQA improvement over prior SOTA under fair comparison.
- Motivation: Region-based visual features are task-specific and can limit the visual semantics available for cross-modal learning.Existing methods commonly use features extracted from object-detection models, whose representations are constrained by visual task design and detection categories.
- Approach: Pixel-BERT combines a CNN-based visual encoder, BERT word-level token embeddings, and multi-modal Transformers to jointly learn visual-language embeddings from pixels and text.The model uses source images rather than bounding-box features as visual input.
- Pre-training: The pre-training procedure uses image-sentence pairs, Masked Language Modeling, and random pixel sampling to learn a universal representation and improve visual robustness.The supplied introduction identifies these components as the main pre-training tasks and mechanism.
- Results: Extensive experiments achieve state-of-the-art performance across VQA, image-text retrieval, and NLVR2.The reported evaluation covers multiple downstream vision-language tasks.
- Results: 2.17 points: Pixel-BERT improves single-model VQA performance over previous SOTA under fair comparison.The passage also reports performance higher than the previous method’s larger model.
2 Related Works
Prior vision-language methods build cross-modal representations with CNN or region-based visual features alongside language representations. Pixel-BERT instead embeds source-image pixels within the cross-modal framework to learn richer visual semantics.
- Existing Architectures: Existing cross-modal methods include two-stream Transformers and architectures that fuse visual and language information through shared Transformer processing.The related-work discussion distinguishes these approaches by network structure.
- Task Context: Cross-modality tasks such as VQA and image captioning require understanding both sentence and visual semantics.
- Visual Representation: Region-based visual features can be limited by the detection categories of Visual Genome, while language contains broader semantic information.Pixel-BERT addresses this mismatch by using source-image input and integrated visual representation learning.
3 Approach
Pixel-BERT is an end-to-end framework that combines visual and sentence embeddings with cross-modality Transformers. It accepts image-sentence pairs, produces joint attention features, and supports MLM and ITM pre-training for downstream vision-language tasks.
- Pixel-BERT uses CNN-based visual encoding, sentence feature embedding, and cross-modal Transformers for joint visual-language representation learning.
- Image-sentence pairs are converted into input elements whose attention features are produced by the model.
- BERT serves as the cross-modality alignment module and models dependencies among all input elements.
- Self-attention and feed-forward operations provide the Transformer’s core computation over visual and language inputs.
- The Transformer builds dense connections within each modality and across image-sentence elements.
3.2 Model Architecture
Pixel-BERT represents language as token embeddings and images through visual inputs rather than relying exclusively on region-based features. Its architecture is designed to jointly process both modalities.
- Vision-language systems tokenize sentences into word vectors and commonly represent images with region-based features from object-detection models.
- Pixel-BERT follows BERT to tokenize words with WordPiece, embed them, and add positional information to the language representation.
Sentence Feature Embedding
The sentence feature embedding follows BERT-style tokenization and positional encoding. The surrounding context contrasts this language pipeline with visual features extracted by region-based detectors.
- Each sentence word is tokenized with WordPiece and embedded into a vector before positional information is added.
- Recent vision-language methods commonly obtain visual features with Faster R-CNN trained on Visual Genome.
Image Feature Embedding
Pixel-BERT learns image representations from CNN-derived pixel features instead of bounding boxes, addressing the category limits of detector-based region features. These visual features are combined with language tokens for cross-modal attention.
- Detector-based region representations are limited by the predefined categories used to extract them.
- Pixel-BERT uses a CNN backbone to extract and flatten spatial image features, representing images through learned pixel features.
- ResNet or ResNeXt backbones with 2 × 2 max pooling down-sample the input image spatially by 64 times in total.
- Pixel features and sentence embeddings are combined with [CLS] and [SEP] tokens as input to the joint-learning Transformer.
Cross-Modality Module
Pixel-BERT combines pixel-based visual representations, BERT word embeddings, and multi-modal transformers in one end-to-end model. It is pre-trained on image-sentence pairs for universal vision-language representation learning.
- The model combines a CNN visual backbone, BERT-based word embeddings, and multi-modal transformers for joint embedding learning.
- The combined CNN and Transformer model is end-to-end trainable, allowing supervision at the Transformer output to update visual representation learning.
- Pixel-BERT uses source images rather than detection-derived region features as input for cross-modality pre-training.
- Masked Language Modeling predicts masked tokens using surrounding text and visual tokens.
Masked Language Modeling
Pixel-BERT uses masked language modeling to connect language tokens with visual content and image-text matching to classify paired versus unpaired inputs. Random pixel sampling additionally improves robustness while reducing computation during pre-training.
- Masked Language Modeling: The MLM objective uses wm for the masked token, θ for model parameters, and P for the generated likelihood.
- Masked Language Modeling: In masked language modeling, masked tokens are inferred from both non-masked language tokens and visual tokens.
- Image-Text Matching: Image-text matching trains a binary classifier on the joint [CLS] representation to distinguish matched from unmatched image-sentence pairs.
- Pixel Random Sampling: Randomly sampling feature pixels encourages learning from incomplete visual input and reduces the number of Transformer input elements.
Pixel Random Sampling
Pixel random sampling is used during pre-training to improve robustness and reduce computation. It is not applied during downstream training or testing because missing information could be harmful and inputs should remain consistent.
- Random pixel sampling is restricted to pre-training rather than downstream tasks.
- Downstream sampling may remove information because fine-tuning lasts only a few epochs.
- Consistent inputs between downstream training and testing provide another reason not to sample pixels downstream.
4 Experiments
Pixel-BERT is evaluated across VQA, NLVR2, and image-text retrieval, with results showing strong performance and benefits from its pre-training design. Experiments also examine pre-training tasks, random pixel sampling, and pixel-language attention.
- Evaluation Setup: Pixel-BERT is evaluated on VQA, NLVR2, and image-text retrieval using datasets including VQA 2.0, NLVR2, Flickr30K, and MS-COCO.
- VQA: 74.45 on test-dev split and 74.55 on test-std split, significantly outperforming existing works on VQA.The result exceeds UNITER (Large), which reports 73.40 on VQA test-std split.
- NLVR2: 76.5 accuracy on dev split and 77.2 accuracy on test-P split, outperforming LXMERT and UNITER in the paired-image setting of NLVR2.The model composes two image-language pairs for this task.
- Image-Text Retrieval: At least 0.6 performance gain on MS-COCO 1K and 0.3 on MS-COCO 5K for image-to-text retrieval recall@1.The experiments report recall@1, recall@5, and recall@10 for both image-to-text and text-to-image retrieval.
- Ablation Study: Both MLM and ITM significantly improve performance on almost all downstream tasks, while ablations evaluate the proposed random pixel sampling method.The ablation study uses downstream VQA, Flickr30K retrieval, and NLVR2 results.
- Attention Analysis: Attention maps associate tokens such as “dog,” “grass,” “frisbee,” and “room” with corresponding image regions without spatial supervision.The visualizations illustrate cross-modality attention learned from image pixels and language.
5 Conclusion and Discussion
Pixel-BERT combines a CNN visual encoder with multi-modal Transformers to align image pixels and text, using pixel sampling and self-supervised pre-training. The model achieves strong downstream performance while avoiding annotated bounding-box restrictions, and the authors identify future work on visual self-supervision and larger datasets.
- Pixel-BERT combines a CNN-based visual encoder with multi-modal Transformers to learn pixel- and text-level visual-language embeddings end to end.
- Random pixel sampling improves robustness, while MLM and ITM pre-train a universal visual-language representation on Visual Genome and MS-COCO.
- The pre-trained model achieves the best performances in most evaluated tasks, including VQA, NLVR2, image-to-text retrieval, and text-to-image retrieval.
- Without annotated bounding-box restrictions, Pixel-BERT can provide stronger image and sentence representations when trained with larger image-sentence pair datasets.
- Future work includes pre-training on Conceptual Captions and designing or combining self-supervised tasks for visual contents.