Source-linked AI summary
Fusion of Detected Objects in Text for Visual Question Answering
Chris Alberti, Jeffrey Ling, Michael Collins, David Reitter
TL;DR
The paper addresses how to encode visual and verbal information together and bind text entities to image regions. It introduces B2T2, which embeds visual features and bounding-box references alongside text tokens, and evaluates it on VCR. B2T2 achieves a more than 25% relative reduction in error rates against the best published and documented model, while ablations identify early visual-textual fusion as critical.
Problem
The paper investigates how visual and linguistic information should be integrated and how text entities should be bound to objects in images for visual question answering.
Method
B2T2 is an early-fusion architecture that embeds visual features and textual references to image bounding boxes at the same level as input word tokens.
Results
More than 25% relative error-rate reduction was achieved on VCR relative to the best published and documented model.
Takeaways & Limitations
Early grounding of words in visual context performs better than late fusion, while incorporating more visual object features and positional features improves performance.
Takeaways & Limitations
The vision model may recognize objects but fail to represent human expressions and activities needed for some VCR questions.
Abstract
from arXiv · showhide
To advance models of multimodal context, we introduce a simple yet powerful neural architecture for data that combines vision and natural language. The "Bounding Boxes in Text Transformer" (B2T2) also leverages referential information binding words to portions of the image in a single unified architecture. B2T2 is highly effective on the Visual Commonsense Reasoning benchmark (https://visualcommonsense.com), achieving a new state-of-the-art with a 25% relative reduction in error rate compared to published baselines and obtaining the best performance to date on the public leaderboard (as of May 22, 2019). A detailed ablation analysis shows that the early integration of the visual features into the text analysis is key to the effectiveness of the new architecture. A reference implementation of our models is provided (https://github.com/google-research/language/tree/master/language/question_answering/b2t2).
1 Introduction
The paper studies how to integrate visual and linguistic context for visual question answering, introducing B2T2 to bind textual references to image regions. On VCR, the model’s early fusion of visual features achieves a substantial error reduction, with ablations identifying early co-reference fusion as especially important.
- VCR requires selecting both an answer and a rationale for image-based questions, with text containing explicit pointers to image bounding boxes.
- The paper asks how textual entities should be bound to image objects and whether multimodal processing should integrate information early or late.
- B2T2 is a neural architecture for multimodal encoding of natural language and images evaluated on the Visual Commonsense Reasoning benchmark.
- More than 25% relative error-rate reduction was achieved against the best published and documented model on VCR.
- Ablations identify early fusion of textual co-references with visual object features as the most critical factor in improving VCR performance.
2 Problem Formulation
The problem formulation assumes each example contains an image, bounding boxes, tokenized text, and a binary label.
- Each example is represented as a 4-tuple (I, B, T, l).
1. I is an image,
The formulation defines the image, bounding-box, text, and label components used to represent each example.
- B is a list of bounding boxes referring to regions of the image.
- T is tokenized text that may include explicit references to elements of B.
- The binary label l indicates whether an answer or rationale choice is correct.
- Bounding-box references are a natural way to discuss image objects, and VCR is annotated with them.
- The image representation function Φ maps an image to a fixed-size vector of dimension d.
- The textual representation maps tokenized text to dimension h using token embeddings and a passage-level representation.
3 Models and Methods
The paper compares late- and early-fusion multimodal architectures, with B2T2 integrating visual features and bounding-box references at the token level. It also describes visual-language pretraining, optimization choices, and VCR ensemble evaluation.
- Architectures: Dual Encoder performs late fusion by separately encoding image and text, then computing answer scores from their representations.Its image representation uses a single visual feature vector, while co-reference information is ignored.
- Architectures: B2T2 performs early fusion by combining text, image, and bounding boxes within non-contextualized token representations.Bounding-box references are encoded through a reference matrix, cropped visual features, and embeddings of box shape and position.
- Architectures: The model represents each bounding box using visual features from its image crop together with learned shape and position information.The visual feature extractor maps each crop to a vector, while π(b_i) supplies the box’s shape and position embedding.
- Pretraining: B2T2 is pretrained on image-caption pairs with impostor identification and masked-language modeling before fine-tuning on VCR.Bounding boxes are absent during pretraining, and the masked-language loss is applied only to positive image-caption pairs.
- Pretraining: Conceptual Captions pretraining slightly improves VCR accuracy while substantially reducing training-result variance.The authors report around 0.4% accuracy improvement and much more stable results across training conditions.
- Implementation and evaluation: The implementation uses fixed ResNet-152 visual features and fine-tuned BERT-Large text representations, with five-seed B2T2 ensembles evaluated on VCR.The ensemble sums class logits from five identical B2T2 models initialized with different random seeds.
4 Data
VCR is a movie-still corpus in which questions, answer choices, and rationales refer to detected image objects through bounding boxes. The benchmark evaluates answer selection, rationale selection, and their sequential combination.
- 4 Data: VCR contains movie stills with questions and answer choices that require selecting one correct answer and one supporting rationale.Each example includes four answers and four rationales, with exactly one correct choice in each set.
- 4 Data: Objects are represented by labeled bounding boxes, and question, answer, and rationale tokens may include deictic references to those objects.The model input appends selected object boxes with class labels and prepends class labels to referenced objects.
- 4 Data: The Q → A task selects the correct answer from four choices given the image, objects, and question.Each VCR example is converted into four instances, one for each answer candidate.
- 4 Data: The QA → R task selects the correct rationale given the image, objects, question, correct answer, and four rationale choices.The pipeline task first selects the answer and then selects a rationale conditioned on that answer.
- 4 Data: The Q → AR task is a two-stage pipeline requiring correct answer selection followed by correct rationale selection.Its success depends on both stages being correct in sequence.
5 Experimental Results
B2T2 outperformed published VCR baselines, while ablations identified bounding boxes and early visual-text fusion as the largest contributors to accuracy. Pretraining mainly reduced training variance, whereas positional information contributed little.
- 5.1 VCR Task Performance: 8.9%, 9.8% and 13.1% absolute accuracy improvements over the published R2C baseline were obtained on Q →A, QA →R and Q →AR, respectively.Both single and ensemble B2T2 models outperformed other systems on the VCR leaderboard as of May 22, 2019.
- 5.2 Ablations: Bounding boxes were the most important ablation factor: accuracy was 67.5% without boxes, 71% with four, and 70.9% with eight but no textual labels.The results show that the model benefits from detected-object information and labels associated with those objects.
- 5.2 Ablations: 3.3% accuracy was lost when visual embeddings were added in BERT’s last layer rather than its first, favoring early fusion.The comparison tests the placement of visual information within the textual representation.
- 5.2 Ablations: 2.9% accuracy was lost when B2T2 used BERT-base instead of BERT-large, indicating that textual model size affected performance.This ablation separates architecture effects from gains attributable to a larger pretrained language model.
- 5.2 Ablations: Pretraining improved accuracy by around 0.4% but heavily reduced variance across learning rates, random seeds and training epochs.The variance reduction was illustrated with a VCR development-set grid search.
- 5.2 Ablations: Removing bounding-box positional embeddings reduced accuracy by only 0.3%, the smallest benefit among the considered ablations.The error analysis also found that object-only visual features were insufficient for recognizing some expressions and activities.
- 5.3 Error Analysis: B2T2 outperformed text-only and dual-encoder models in examples where combining information from image regions could provide scene-level clues.Other examples exposed failures shared by B2T2 and the dual encoder when image information about expressions or activities was unavailable.
6 Related Work
Related work spans multimodal language models, region-based visual question answering, and explicit reasoning architectures. B2T2 differs by jointly representing textual and visual tokens while retaining bounding-box references.
- Multimodal Transformers: VideoBERT jointly processes video frames and text tokens with a Transformer, but it cannot answer questions and does not model bounding boxes.This positions B2T2 as a question-answering architecture with explicit object-region information.
- Region-Based VQA: Bottom-Up Top-Down attention attends from text to detected object boxes, whereas its top-down component attends to regular image-grid regions.B2T2 is similar in attending to preprocessing-generated boxes but uses a joint token representation.
- Grounded Concepts: The Neuro-Symbolic Concept Learner uses a semantic parser for visually grounded concepts, while B2T2 uses a Transformer to construct a joint representation of textual and visual tokens.The distinction is between parser-mediated grounding and Transformer-based joint encoding.
- Reasoning Architectures: MAC uses explicit multi-step reasoning without bounding boxes and reduces textual information to a fixed-length vector representation.It was reported to outperform Bottom-Up Top-Down on GQA.
- Subsequent Work: Subsequent systems including ViLBERT, VL-BERT, Unicoder-VL and VisualBERT exceeded B2T2’s performance in some cases.These works appeared after the paper’s submission.
- VQA Datasets: VCR differs from other visual question answering datasets through more complex language, adversarially selected answer choices, or non-synthetic scenes, depending on the dataset comparison.The cited comparison includes VQA, CLEVR and GQA.
7 Conclusion
The paper compares late- and early-fusion multimodal architectures, finding that B2T2’s early integration of visual features with text achieves the strongest results and that deeper visual incorporation and pretraining further improve performance.
- B2T2 embeds visual features alongside input word tokens, linking sentences to image bounding boxes in an early-fusion architecture.
- The early-fusion B2T2 model produces the best available results in visual question answering tasks.
- A late-fusion control model performs substantively worse than B2T2, supporting early rather than late grounding of words in visual context.
- Deeply incorporating visual features from the entire image and bounding boxes substantially improves the Dual Encoder model.
- Conceptual Captions pretraining with a Mask-LM loss provides a small additional improvement and more stable fine-tuning results.