Source-linked AI summary
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, Fei Huang
TL;DR
Detector-based VLP pipelines rely on task-specific region features and can be computationally inefficient. E2E-VLP jointly learns visual representations and image-text alignment with object detection and captioning in an end-to-end encoder-decoder. The paper reports strong performance across vision-language tasks and reduced online inference time.
Problem
Most VLP methods use a two-stage pipeline with detector-derived visual features, which the paper describes as task-specific and inefficient.
Source: §1 · p. 1Method
E2E-VLP jointly learns visual representations and image-text alignment end-to-end, incorporating object detection and image captioning in a unified Transformer encoder-decoder.
Source: §1 · p. 2Results
E2E-VLP reports strong performance across vision-language tasks and greatly decreased online inference time with its one-stage solution.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Vision-language pre-training (VLP) on large-scale image-text pairs has achieved huge success for the cross-modal downstream tasks. The most existing pre-training methods mainly adopt a two-step training procedure, which firstly employs a pre-trained object detector to extract region-based visual features, then concatenates the image representation and text embedding as the input of Transformer to train. However, these methods face problems of using task-specific visual representation of the specific object detector for generic cross-modal understanding, and the computation inefficiency of two-stage pipeline. In this paper, we propose the first end-to-end vision-language pre-trained model for both V+L understanding and generation, namely E2E-VLP, where we build a unified Transformer framework to jointly learn visual representation, and semantic alignments between image and text. We incorporate the tasks of object detection and image captioning into pre-training with a unified Transformer encoder-decoder architecture for enhancing visual learning. An extensive set of experiments have been conducted on well-established vision-language downstream tasks to demonstrate the effectiveness of this novel VLP paradigm.
1 Introduction
Existing detector-based VLP pipelines use task-specific visual features and incur efficiency costs. E2E-VLP jointly learns visual representations and image–text alignment with fine-grained visual tasks, and reports strong performance across vision-language tasks with reduced online inference time.
1 Introduction
- Earlier two-stage VLP systems use detector-derived, task-specific visual features that may miss important information and add feature-extraction and prediction-time costs.Cached visual features also constrain model design.Source: §1 · p. 1
- Prior end-to-end grid-feature work omitted fine-grained visual pre-training tasks, which the paper identifies as important for vision-language pre-training.The introduction notes that object detection and image captioning are incorporated to enhance visual learning.Source: §1 · pp. 1–2
- E2E-VLP jointly learns visual representations and image–text alignment in an end-to-end encoder-decoder, incorporating object detection and image captioning for fine-grained visual learning.The paper describes detection as learning object-level semantics and captioning as capturing text-aligned visual semantics.Source: §1 · p. 2
- E2E-VLP reports comparable or superior performance and faster online inference, with strong results across visual question answering, visual reasoning, retrieval, and captioning.The paper presents this as an outcome of its one-stage pre-training paradigm.Source: §1 · p. 2
2 Related Work
Prior VLP work includes single-stream and two-stream Transformer pipelines that commonly rely on cached detector features. E2E-VLP addresses the absence of object-level visual tasks in prior end-to-end grid-feature pre-training by incorporating detection and captioning.
2 Related Work
- Existing VLP methods include single-stream and two-stream Transformers, while cached detector features constrain model design and complicate end-to-end pre-training.The paper describes Pixel-BERT as the first end-to-end grid-feature approach, but one without object-level visual tasks.Source: §2 · pp. 2–3
- Pixel-BERT's end-to-end grid-feature pre-training omits object-level visual tasks, which the paper says are important for aligning cross-modal representations.Source: §2 · p. 3
- E2E-VLP incorporates object detection and image captioning into pre-training to enhance fine-grained visual-language understanding and generation.Its unified encoder-decoder jointly learns visual representations and image-text semantic alignment.Source: §2 · p. 3, Figure 1 · p. 3
3 E2E-VLP Pre-training
E2E-VLP jointly learns visual representations and cross-modal alignment in an end-to-end Transformer encoder-decoder. It combines encoder-side language-image tasks with decoder-side object detection and caption generation, training the tasks jointly.
3.1 Model Architecture
- A CNN image encoder and Transformer encoder-decoder are trained end to end to learn visual representations and high-level cross-modal alignment.The framework uses a CNN backbone to extract representations from pixels and a Transformer for cross-modal learning.Source: §3.1 · p. 3
3.1.1 Input Representations
- Pixel features replace detector-derived representations, are flattened into a sequence with positional encodings, and are concatenated with text embeddings for cross-modal attention.The encoder input sequence combines text embeddings with image grid features.Source: §3.1.1 · p. 4, §3.1.2 · p. 4
- The encoder uses image features to help predict masked words, resolving ambiguity, and learns cross-modal attention over image grid features and language tokens.Image-Text Matching is also among the encoder-side pre-training tasks.Source: §3.1.2 · p. 4
3.1.3 Visual-enhanced Decoder
- The decoder shares attention parameters across object detection and captioning, pairing fine-grained object understanding with visual learning guided by textual semantics.Detection is formulated as direct set prediction with bipartite matching; captioning generates text autoregressively from image context.Source: §3.1.3 · pp. 4–5
3.2 Joint Training
- Joint pre-training minimizes losses for Masked Language Modeling, Image-Text Matching, Object Detection, and Image-to-Text Generation.The object detection task also includes attribute prediction for Visual Genome, alongside class prediction and box regression.Source: §3.2 · p. 5, §3.1.3 · p. 5
4 Experiments
E2E-VLP is pre-trained on MS-COCO and Visual Genome using image-text and object-related annotations. The implementation uses 6- or 12-layer Transformer encoders and a fixed 6-layer decoder.
4.1 Pre-training Dataset
- The pre-training data comprise 6.01M image-and-sentence pairs from 180K distinct images in MS-COCO and Visual Genome.The datasets provide object detection annotations; MS-COCO also provides image captions, while Visual Genome provides region descriptions.Source: §4.1 · p. 5
4.2 Implementation Details
- The model uses 6- or 12-layer Transformer encoders, a fixed 6-layer decoder, and 256 hidden units per layer block.Training runs for 200 epochs with total batch size 32 on 8 V100 GPUs; input images are resized with the shortest side between 480 and 800 pixels and longest side at most 1333.Source: §4.2 · p. 5
5 Experiments
The experiments compare E2E-VLP with similarly sized VLP models and evaluate it on downstream understanding, reasoning, retrieval, and generation tasks. The supplied passages describe the datasets and fine-tuning setups, but provide no numerical outcomes.
5 Experiments
- E2E-VLP is compared with competitive VLP models of comparable size across downstream vision-language tasks.The described task set includes VQA, NLVR2, image captioning, and image-text retrieval.Source: §5.1 · p. 5
- The evaluation tables cover VQA, NLVR2, and image captioning in Table 1, and Flickr30K in Table 2.The supplied passages identify the tables’ scope but do not provide their result values.Source: Table 1 · p. 6, Table 2 · p. 6
- VQA uses 204K images and 1.1M questions, with answers selected from a shared set of 3,129 classes.The supplied setup treats VQA as multi-label classification and trains for 12 epochs.Source: §5.1 · p. 6
- NLVR2 tests whether a statement is true of a pair of images, while captioning trains the model to generate image descriptions.Captioning uses a seq2seq loss and is evaluated on COCO.Source: §5.1 · p. 6
- Flickr30K image-text retrieval is formulated as a ranking problem with image retrieval and text retrieval as its two directions.The dataset contains 31,000 images, each with five captions.Source: §5.1 · pp. 6–7
5.2 Baseline Methods
The baselines span detector-based single-stream and two-stream VLP models, alongside the end-to-end PixelBERT model. The comparison includes methods with different visual-input and fusion designs.
5.2 Baseline Methods
- The comparison includes single-stream and two-stream architectures using region-based visual features extracted by an object detector.Single-stream examples include OSCAR, UNITER, Unicoder-VL, VLBERT, and VLP; two-stream examples include ERNIE-VIL, LXMERT, and ViLBERT.Source: §5.2 · p. 7
- Single-stream models encode vision-language inputs with a unified Transformer, whereas two-stream models encode image and text separately before fusion.The listed examples are OSCAR, UNITER, Unicoder-VL, VLBERT, and VLP for single-stream; ERNIE-VIL, LXMERT, and ViLBERT for two-stream.Source: §5.2 · p. 7
- PixelBERT is the end-to-end baseline and uses random pixel sampling without visual semantic understanding pre-training tasks.The passage identifies its lack of such tasks as a distinction from E2E-VLP’s approach.Source: §5.2 · p. 7
5.3 Main Results
E2E-VLP achieves comparable performance to two-step region-feature methods despite using fewer parameters and only in-domain pre-training data, and significantly improves on PixelBERT. The supplied results also identify an ablation table for visual pre-training tasks.
5.3 Main Results
- With fewer parameters and only in-domain pre-training data, E2E-VLP achieves comparable performance to two-step region-feature methods such as OSCAR and ERNIE-VIL.The passage describes this as evidence for the end-to-end grid-feature approach.Source: §5.3 · p. 7
- Table 3 reports development-set ablations of visual pre-training tasks using a six-layer encoder and ResNet50 backbone.The supplied table caption does not state the ablation outcomes.Source: Table 3 · p. 7
- E2E-VLP significantly improves upon PixelBERT, an end-to-end baseline, in the reported comparison.The passage connects the advantage to fine-grained visual learning with object detection and image captioning.Source: §5.3 · p. 7
5.4 Importance of Visual Learning
Ablations show contributions from all three visual pre-training tasks, with object detection and attribute prediction more important than image-to-text generation on VQA and NLVR2. E2E-VLP’s end-to-end inference is faster than two-step models while retaining comparable results on those tasks.
5.4 Importance of Visual Learning
- Removing any of the three visual pre-training tasks decreases performance on both VQA and NLVR2.Object detection and attribute prediction capture fine-grained object semantics, while image-to-text generation guides visual-feature learning with textual semantics.Source: §5.4 · p. 7
- Object detection and attribute prediction matter more than image-to-text generation for VQA and NLVR2, which focus more on fine-grained object semantics.The passage presents this as a possible explanation for the relative importance.Source: §5.4 · p. 7
5.5 Inference Efficiency
- Table 5 reports development-set results for different pre-trained model architectures.The supplied table caption identifies its scope but not its result values.Source: Table 5 · p. 8
- End-to-end methods are 2–3 times faster online than two-step models, and E2E-VLP achieves comparable VQA and NLVR2 results while saving about 3.5 times running time.The comparison examines average inference time per query; about 80% of the two-step models’ 500 ms per image-text pair is spent extracting region features with Faster R-CNN.Source: §5.5 · pp. 7–8, Table 4 · p. 8
- E2E-VLP also gains some speed over PixelBERT, which the passage attributes to its smaller Transformer hidden size of 256.The passage also reports that E2E-VLP improves performance upon PixelBERT in connection with its visual pre-training tasks.Source: §5.5 · p. 8
5.6 Architecture Selection
Both Transformer depth and visual-backbone strength contribute to performance, while input image size trades inference speed against accuracy.
5.6 Architecture Selection
- The architecture study varies Transformer encoder depth and ResNet visual-backbone layers to assess their contributions to cross-modal understanding.The authors examine whether the visual backbone or Transformer network matters more for cross-modal understanding and fusion.Source: §5.6 · p. 8
- A stronger visual backbone such as ResNet-152 provides more benefit than increasing Transformer encoder layers from 6 to 12.Adding encoder layers and using more complex backbones both contribute to performance.Source: §5.6 · p. 8
5.7 Impact of Input Image Size
- Reducing input size from (800, 1333) to (448, 448) makes inference about 5 times faster, with performance decreasing about 2%-3%.Larger images benefit performance but produce longer visual sequences.Source: §5.7 · p. 8
5.8 Object Detection with Paired Text
E2E-VLP is fine-tuned for object detection on paired image-text inputs, and the authors report strong performance against the original DETR model.
5.8 Object Detection with Paired Text
- Table 7 reports object-detection results on the MSCOCO development dataset.Source: Table 7 · p. 9
- E2E-VLP is fine-tuned on paired image-text inputs for MSCOCO object detection, using a decoder as in DETR.The authors use this evaluation to examine cross-modal fusion and fine-grained semantics.Source: §5.8 · p. 9
- E2E-VLP supports object detection based on text-image pairs and performs surprisingly well compared with the original DETR model.The authors suggest this may demonstrate fine-grained semantic capture and multimodal fusion for a visual-only task.Source: §5.8 · p. 9
6 Conclusion
E2E-VLP jointly learns visual representations and image-text semantic alignments from pixels in an end-to-end framework. The authors report effective and efficient downstream performance and identify deeper fusion and advanced pre-training tasks as future directions.
6 Conclusion
- E2E-VLP uses image grid features rather than region features from a two-stage pipeline, jointly learning visual representations and image-text semantic alignments.The unified Transformer encoder-decoder incorporates object detection and image captioning tasks to enhance visual learning.Source: §6 · p. 9
- Experiments on established vision-language downstream tasks demonstrate E2E-VLP's effectiveness and efficiency.Source: §6 · p. 9
- Future work will explore more deeply interacted image-text fusion and incorporate more advanced vision-language pre-training tasks.Source: §6 · p. 9