Source-linked AI summary
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang
TL;DR
Vision-language pre-training methods often fail to distinguish common words from fine-grained semantic concepts needed to interpret scenes. ERNIE-ViL incorporates scene-graph knowledge through object, attribute, and relationship prediction tasks, achieving state-of-the-art results on five downstream tasks and first place on VCR with a 3.7% absolute Q→AR improvement.
Problem
Existing vision-language pre-training methods neglect detailed semantic alignments involving objects, attributes, and relationships across vision and language.
Method
ERNIE-ViL parses text into scene graphs and adds Object Prediction, Attribute Prediction, and Relationship Prediction tasks during cross-modal pre-training.
Results
ERNIE-ViL achieves state-of-the-art performance on all 5 downstream cross-modal tasks and ranks first on VCR, with a 3.7% absolute Q→AR improvement.
Takeaways & Limitations
Incorporating structured scene-graph knowledge improves joint vision-language representations by emphasizing detailed semantic alignments.
Abstract
from arXiv · showhide
We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7%.
Introduction
Existing vision-language pre-training methods underrepresent fine-grained semantics, while ERNIE-ViL introduces scene-graph prediction tasks to align objects, attributes, and relationships across modalities. It reports state-of-the-art performance across five downstream tasks, including a 3.7% absolute improvement on VCR Q→AR.
- Motivation: Existing methods use randomly masked sub-words, masked regions, and image-text matching, but do not distinguish common words from detailed semantic words.The neglected semantics include objects, object attributes, and relationships between objects.
- Motivation: Detailed semantics determine scene interpretation, including differences in objects, attributes, and relationships.Figure 1 contrasts scene pairs involving dogs versus cats, purple versus green attributes, and rides versus repairing relationships.
- Results: 3.7% absolute improvement on VCR Q→AR over the state-of-the-art performance accompanies first place on the VCR leaderboard.ERNIE-ViL also achieves state-of-the-art performance on all five downstream cross-modal tasks.
- Contribution: ERNIE-ViL introduces structured knowledge into vision-language pre-training through scene graphs.The paper identifies this as its first work contribution to the best of its knowledge.
- Method: ERNIE-ViL constructs Scene Graph Prediction tasks during pre-training to emphasize cross-modal detailed semantic alignments.The approach is inspired by structured knowledge masking and predicts different scene-graph node types.
Related Works
Prior vision-language pre-training research varies in architecture, tasks, and data, while scene graphs provide structured fine-grained information about visual scenes. ERNIE-ViL builds on this direction by using scene-graph information for cross-modal semantic alignment.
- Research dimensions: Cross-modal pre-training models primarily differ in model architecture, pre-training tasks, and pre-training data.These dimensions organize the prior research discussed in the paper.
- Model Architecture: Architectures include uniform cross-modal Transformers and two-stream Transformers with separate image and text representations.ViLBERT and LXMERT are cited as examples of the two-stream design.
- Pre-training Tasks: Common pre-training tasks include Masked Language Modeling, Masked Region Prediction, and Image-Text Matching.The paper notes that random sub-word masking does not distinguish common words from detailed semantic words.
- Pre-training Data: Conceptual Captions and SBU Captions provide widely used out-of-domain image-description data, with 3.0M and 1.0M pairs respectively.Existing work also incorporates in-domain datasets such as MS-COCO and Visual Genome.
- Scene Graphs: Scene graphs encode objects, object attributes, and relationships, providing structured fine-grained semantic information for vision-language modeling.Scene-graph information has supported state-of-the-art models in image captioning, retrieval, VQA, and image generation.
- Scene Graph Prediction: ERNIE-ViL uses Object Prediction, Attribute Prediction, and Relationship Prediction tasks based on scene graphs parsed from text.These tasks are illustrated as mechanisms for learning detailed semantic alignments across image and text modalities.
Approach
ERNIE-ViL uses scene-graph knowledge and two-stream cross-modal Transformers to learn fine-grained alignments between visual regions and textual semantics. Its Scene Graph Prediction tasks mask and recover objects, attributes, and relationships using language context and image regions.
- Model Architecture: ERNIE-ViL uses two-stream cross-modal Transformers to jointly model intra-modal and inter-modal representations from sentence and image inputs.Parallel image and text encoders are cross-attended through cross-modal Transformer blocks.
- Scene Graph Prediction: Scene Graph Prediction masks and predicts different node types to emphasize detailed semantic alignments across vision and language.The tasks include Object Prediction, Attribute Prediction, and Relationship Prediction.
- Scene Graph Prediction: Scene graphs represent objects, object attributes, and relationships, providing structured knowledge for fine-grained vision-language representations.The parsed graph is denoted G(w)=<O(w), E(w), K(w)>.
- Object Prediction: Object Prediction recovers masked object tokens from surrounding words and all image regions, linking visual content to object-level semantics.Object nodes are selected for masking and their corresponding text subsequences are masked.
- Attribute Prediction: Attribute Prediction masks attribute nodes while retaining associated objects, then recovers attributes using object tokens, surrounding words, and image regions.Attributes encode fine-grained visual information such as object color or shape.
- Relationship Prediction: Relationship Prediction masks relationship nodes in object triplets and recovers them from the two object tokens, surrounding text, and all image regions.Relationships encode semantic actions or relative positions that help distinguish scenes with identical objects.
Training ERNIE-ViL
ERNIE-ViL is pre-trained on large automatically collected image-caption datasets with region, token, and scene-graph masking objectives. Training uses object-detection regions, negative image-text pairs, and two model-size settings.
- Pre-training Data: Conceptual Captions and SBU Captions provide approximately 3.0 million and 0.8 million available image-caption pairs for out-of-domain pre-training.The original datasets contain 3.3 million and 1.0 million pairs, respectively, but broken links reduce the usable counts.
- Input Preparation: Faster R-CNN selects 10 to 36 salient image regions whose mean-pooled convolutional representations serve as region features.Regions are retained when class detection probability exceeds 0.2.
- Masking and Matching: Training randomly masks 15% of text tokens, 30% of scene-graph nodes, and 15% of image regions.Negative image-text pairs are formed by randomly selecting an image for each text.
- Optimization: ERNIE-ViL-base and ERNIE-ViL-large mainly differ in text-stream depth and are trained with cross-transformers on 8 V100 GPUs for 700k steps.The total batch size is 512, with Adam optimization and an initial learning rate of 1e-4.
Downstream Tasks
ERNIE-ViL is evaluated on VQA, VCR, referring-expression grounding, and image- and text-retrieval tasks. These evaluations use task-specific datasets and matching or classification formulations.
- Visual Commonsense Reasoning: VCR includes multiple-choice visual question answering, answer justification, and holistic Q→AR evaluation requiring both answer and rationale correctness.The two subtasks are Q→A and QA→R.
- Visual Question Answering: VQA 2.0 evaluates image-based question answering as multi-label classification using soft target scores derived from ten human answers.The dataset contains 204k images and 1.1M questions.
- Referring Expression Grounding: RefCOCO+ evaluates referring-expression grounding by localizing the image region corresponding to a natural-language reference.A region is positive when its IoU with the ground-truth box exceeds 0.5.
- Image Retrieval & Text Retrieval: Flickr30K retrieval identifies an image from a caption-described pool using image-text matching scores.Flickr30K contains 31,000 images with five captions per image.
- Evaluation Setup: Table 3 compares ERNIE-ViL with previous pretrained models across downstream vision-language tasks, including image retrieval and text retrieval.The table distinguishes IR from TR and reports special VCR test results for models pretrained on out-of-domain data.
Results
ERNIE-ViL outperforms comparable vision-language pre-training methods across five downstream tasks, including state-of-the-art performance when trained with both out-of-domain and in-domain data.
- ERNIE-ViL obtains the best performances on all 5 downstream tasks among methods pre-trained on the same out-of-domain datasets.The comparison uses Conceptual Captions and SBU Captions.
- 6.60% improvement on VCR (Q→AR) and 1.74% on VQA (test-std) are achieved by ERNIE-ViL-large over VLBERT-large.
- 2.40% improvement is obtained on both testA and testB splits of RefCOCO+ compared with VLBERT-large.
- 2.94% improvement on R@1 for image retrieval and 0.50% on R@1 for text retrieval are achieved over Unicoder-VL-base.
- ERNIE-ViL-large achieves state-of-the-art performance when compared with models pretrained on both out-of-domain and in-domain datasets.The comparison includes existing works such as UNITER, OSCAR, and VILLA.
Analysis
Analyses attribute ERNIE-ViL’s gains to Scene Graph Prediction tasks, which improve detailed-semantic alignment in downstream evaluation and cloze prediction.
- Effectiveness of Scene Graph Prediction tasks: Pre-training with SGP tasks significantly improves performance across all downstream tasks.
- Effectiveness of Scene Graph Prediction tasks: 0.69% accuracy improvement on RefCOCO+ and 2.22% R@1 improvement for image retrieval are observed with SGP tasks.
- Effectiveness of Scene Graph Prediction tasks: ERNIE 2.0 text initialization further improves all tasks and produces a relatively large improvement on VCR.The paper attributes this benefit to common sense knowledge learned through continual pre-training on various tasks.
- Cloze Test: 2.0% overall ACC@1 improvement is reported in the cloze test between models with and without SGP tasks.
- Cloze Test: 3.08% improvement for relationships, 1.84% for attributes, and 1.20% for objects are achieved on ACC@1 with SGP tasks.
- Cloze Test: Without SGP tasks, predictions fail to distinguish common words from detailed-semantic words in some cases, while SGP-trained models can predict reasonable tokens with higher confidence.
Conclusion
ERNIE-ViL enhances vision-language joint representations by adding Scene Graph Prediction tasks to conventional cross-modal pre-training.
- ERNIE-ViL introduces Scene Graph Prediction tasks to characterize detailed semantic alignments across vision and language.The conclusion also identifies incorporating structured knowledge from scene graphs as beneficial in downstream experiments.
- Future work could incorporate scene graphs extracted from images and use Graph Neural Networks to integrate more structured knowledge.