PaperlayerView exact revision on arXiv

Source-linked AI summary

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang

arXiv:1707.07998v3cs.CV

TL;DR

Image captioning and VQA need fine-grained visual attention, but conventional methods give little consideration to how attended regions are determined. This paper combines bottom-up region proposals with top-down task-specific weighting, achieving state-of-the-art results in both tasks, including a CIDEr score of 117.9 and first place in the 2017 VQA Challenge.

  • Problem

    Existing captioning and VQA attention mechanisms typically focus on CNN features while giving little consideration to how attended image regions are determined.

  • Method

    The paper combines bottom-up proposals for salient image regions with top-down, task-specific attention weighting over those regions.

  • Results

    The approach achieves state-of-the-art results in both image captioning and VQA, including a CIDEr score of 117.9 and first place in the 2017 VQA Challenge.

  • Takeaways & Limitations

    The approach more closely unifies visual-language understanding with object detection, with immediate benefits obtainable by replacing pretrained CNN features with pretrained bottom-up attention features.

  • Takeaways & Limitations

    Important aspects of the VQA approach are not detailed, with full model specifics and hyperparameters deferred to another work.

Abstract

from arXiv · show

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

1. Introduction

The paper argues that object- and saliency-level attention is a more natural basis for image captioning and VQA than attention over uniform CNN grids. It therefore combines bottom-up region proposals with top-down, task-specific weighting and reports strong image-captioning results.

  • Image captioning and VQA often require fine-grained visual processing and multiple reasoning steps, motivating widespread use of visual attention mechanisms.
  • Top-down attention uses task-specific context to select CNN features, but conventional approaches largely ignore the content of the image regions being attended to.
  • Uniform grids of equally sized receptive fields are less natural for attention than objects and other salient image regions.
  • The proposed mechanism combines bottom-up Faster R-CNN region proposals and pooled feature vectors with top-down prediction of an attention distribution over those regions.
  • 117.9 CIDEr, 21.5 SPICE, and 36.9 BLEU-4 were achieved on the MSCOCO test server, establishing a new state-of-the-art for image captioning.

2. Related Work

Prior attention-based models for image captioning and VQA are typically top-down, conditioning spatial CNN attention on partial captions or questions. Comparatively few approaches attend directly to salient image regions, using region proposals or specialized feature-generation methods.

  • Top-down attention: Attention-based deep networks typically use top-down attention over CNN-layer outputs, conditioned on a partial caption for image captioning or a question representation for VQA.These models predict a weighting for each spatial location.
  • Salient-region attention: Comparatively few previous works apply attention to salient image regions.The passage identifies two such papers.
  • Salient-region attention: Jin et al. identify salient regions with selective search, filter them with a classifier, resize and CNN-encode them, then provide them to an attention-based captioning model.This describes one prior salient-region approach for image captioning.
  • Salient-region attention: The Areas of Attention model generates image features with edge boxes or spatial transformer networks and processes them using an attention model based on three bilinear pairwise interactions.This describes the second prior salient-region approach.

3. Approach

The approach represents images as salient-region features proposed by Faster R-CNN and applies soft top-down attention during caption generation. The captioning model uses two LSTM layers, while training combines cross-entropy with CIDEr-oriented sequence optimization.

  • Image features: Image features V are a variably sized set of k region vectors, each encoding a salient image region.The features may come from the bottom-up attention model or a CNN spatial output layer.
  • Bottom-up attention: Faster R-CNN with ResNet-101 selects bounding-box regions whose mean-pooled convolutional features form 2048-dimensional vectors.Regions are retained when any class detection probability exceeds a confidence threshold, making the mechanism a hard attention model.
  • Captioning model: The captioning model uses a first LSTM for top-down visual attention and a second LSTM as the language model.The attention LSTM receives language-LSTM output, the mean-pooled image feature, and the previously generated word encoding.
  • Captioning model: At each time step, normalized attention weights are generated for all k image features, and their convex combination is passed to the language LSTM.The attended image feature is concatenated with the attention-LSTM output before language modeling.
  • Training: Training minimizes cross-entropy and can subsequently optimize the expected CIDEr score using Self-Critical Sequence Training.The CIDEr optimization is initialized from the cross-entropy-trained model and is completed in a single epoch.

4. Evaluation · 4.1. Datasets

The evaluation uses Visual Genome for bottom-up attention pretraining and VQA augmentation, MSCOCO for captioning, and VQA v2.0 for visual question answering. Dataset splits, preprocessing, vocabularies, and evaluation protocols are specified for each task.

  • 4.1.1 Visual Genome Dataset: Visual Genome contains 108K densely annotated images and 1.7M visual question answers, supporting bottom-up attention pretraining and VQA data augmentation.Annotations include objects, attributes, relationships, and question-answer pairs.
  • 4.1.1 Visual Genome Dataset: For bottom-up attention pretraining, Visual Genome uses 98K training images, with 5K each reserved for validation and future testing.Overlapping Visual Genome and MSCOCO images are kept in matching splits to prevent contamination.
  • 4.1.1 Visual Genome Dataset: The Up-Down captioning model achieves 3–8% relative gains across all metrics over the ResNet baseline, under both cross-entropy and CIDEr optimization.The comparison is made on the MSCOCO Karpathy test split.
  • 4.1.1 Visual Genome Dataset: Cleaning reduces Visual Genome annotations from 2,000 object classes and 500 attribute classes to 1,600 object classes and 400 attribute classes.Abstract classes with poor initial detection performance are manually removed, while overlapping classes remain separate.
  • 4.1.2 Microsoft COCO Dataset: MSCOCO 2014 provides 113,287 training images and 5K validation and test images in the Karpathy splits, while the test-server model uses all 123K training and validation images.Caption evaluation uses SPICE, CIDEr, METEOR, ROUGE-L, and BLEU.
  • 4.1.2 Microsoft COCO Dataset: The captioning vocabulary contains 10,010 words after lowercasing, whitespace tokenization, and filtering words occurring fewer than five times.These preprocessing choices follow standard practice.
  • 4.1.3 VQA v2.0 Dataset: VQA v2.0 contains 1.1M questions and 11.1M answers about MSCOCO images, with questions trimmed to 14 words and a 3,129-answer output vocabulary.Training incorporates eligible Visual Genome question-answer pairs, and answer quality is evaluated by accuracy.

4.2. ResNet Baseline

The experiments compare the full Up-Down model with prior work and a ResNet ablation to quantify bottom-up attention’s impact in captioning and VQA. The ResNet baseline replaces bottom-up attention with ImageNet-pretrained CNN image features.

  • 4.2. ResNet Baseline: The ResNet baseline replaces bottom-up attention with a ResNet CNN pretrained on ImageNet to encode each image.It is evaluated against the full Up-Down model and prior work in both captioning and VQA experiments.
  • 4.2. ResNet Baseline: In image captioning, ResNet-101’s final convolutional layer produces a fixed 10×10 spatial representation from the full-sized input image.Bilinear interpolation resizes the convolutional output, matching the maximum number of spatial regions used in the full model.
  • 4.2. ResNet Baseline: In VQA experiments, the resized input image is encoded with ResNet-200.The passage also notes separate experiments evaluating the effect of varying spatial representation size.

4.3. Image Captioning Results

The full Up-Down model substantially improves image-captioning performance over the ResNet baseline and SCST across training objectives and metrics. It achieves the best reported single-model Karpathy-split results and outperforms prior MSCOCO test-server submissions on all reported metrics.

  • Evaluation setup: Models are evaluated on the Karpathy test split using standard cross-entropy or CIDEr-optimized training, with single models and no input ResNet/R-CNN fine-tuning.SCST uses ResNet-101 full-image encoding and selects its results from four random initializations.
  • Baseline comparison: The ResNet baseline slightly exceeds SCST with cross-entropy loss but slightly trails it when optimized for CIDEr.The comparison uses the same reported evaluation setting described for the Karpathy splits.
  • Bottom-up attention: The full Up-Down model significantly improves all metrics under both cross-entropy and CIDEr optimization, achieving the best reported single-model Karpathy test-split results.The improvement follows incorporation of bottom-up attention.
  • Bottom-up attention: Bottom-up attention contributes to identifying objects, object attributes, and relationships between objects.These capabilities are described as the contribution illustrated in the accompanying results discussion.
  • MSCOCO evaluation: On the official MSCOCO evaluation server, four CIDEr-optimized ensembled models outperform all other test-server submissions on every reported evaluation metric.This ranking reflects the submission status on 18 July 2017.

4.4. VQA Results · 4.5. Qualitative Analysis

The Up-Down VQA model improves over ResNet baselines across all question types, while qualitative examples show attention operating over fine details, large regions, and object-level visual concepts. Its 30-model ensemble is evaluated on the official VQA 2.0 test-standard server.

  • 4.4. VQA Results: Bottom-up attention significantly improves the full Up-Down VQA model over the best ResNet baseline across all question types.The comparison uses the VQA v2.0 validation set, although the ResNet baseline has approximately twice as many convolutional layers.
  • 4.4. VQA Results: The VQA results include performance from 30 ensembled models on the official VQA 2.0 test-standard evaluation server.
  • 4.4. VQA Results: An attention example answers “kitchen” to “What room are they in?” by focusing on the stovetop.
  • 4.5. Qualitative Analysis: The Up-Down captioning model can focus on either fine details or large image regions.Figure 5 visualizes attended regions for different words generated by the model.
  • 4.5. Qualitative Analysis: This flexibility arises from overlapping attention candidates spanning varied scales and aspect ratios.Candidates align to an object, several related objects, or another salient region.
  • 4.5. Qualitative Analysis: Object-level candidate regions spatially co-locate associated visual concepts, allowing the model to process information about an object together.The passage contrasts this with conventional approaches and describes it as a natural implementation of attention.

5. Conclusion

The paper presents a combined bottom-up and top-down attention mechanism that operates naturally over objects and salient regions, achieving state-of-the-art results in image captioning and VQA while improving attention interpretability. It also unifies visual-linguistic tasks with object detection and suggests replacing pretrained CNN features with pretrained bottom-up attention features.

  • Conclusion: The proposed mechanism calculates attention naturally at the level of objects and other salient image regions.It combines bottom-up and top-down visual attention.
  • Conclusion: The approach achieves state-of-the-art results in both image captioning and visual question answering.The resulting attention weights are also more interpretable.
  • Conclusion: The work more closely unifies visual and linguistic understanding with recent progress in object detection.This connection suggests several directions for future research.
  • Conclusion: A practical immediate benefit is replacing pretrained CNN features with pretrained bottom-up attention features.The paper presents this replacement as a straightforward way to capture the approach’s benefits.

SUPPLEMENTARY MATERIALS · 6. Implementation Details

The supplementary implementation details specify region-selection settings for bottom-up attention and the core training configurations for the captioning and VQA models. These choices include detection thresholds, model dimensions, optimization schedules, regularization, and training hardware and duration.

  • 6.1. Bottom-Up Attention Model: Bottom-up attention uses IoU thresholds of 0.7 for region proposal suppression and 0.3 for object class suppression.A class detection confidence threshold of 0.2 selects salient regions, with up to 100 regions per image.
  • 6.1. Bottom-Up Attention Model: The number of selected regions varies with image complexity, up to a maximum of 100 regions per image.Initial experiments found that selecting the top 36 features per image performed almost as well in both downstream tasks.
  • 6.2. Captioning Model: The captioning model uses LSTM hidden size M = 1,000, attention-layer hidden size H = 512, and word-embedding size E = 1,000.These settings define the principal dimensions of the captioning architecture.
  • 6.2. Captioning Model: Captioning training begins at learning rate 0.01 and decreases linearly to zero over 60K iterations with batch size 100 and momentum 0.9.Training on two Nvidia Titan X GPUs takes around 9 hours.
  • 6.3. VQA Model: The VQA model uses 300 dimension word embeddings initialized with pretrained GloVe vectors and hidden states of dimension 512.The model is trained with AdaDelta and regularized using early stopping.
  • 6.3. VQA Model: VQA model training takes in the order of 12–18 hours on a single Nvidia K40 GPU.The implementation refers readers to Teney et al. for further details.

7. Additional Examples

Qualitative examples show that the Up-Down model attends to relevant objects and fine details, avoiding errors made by the ResNet baseline in unusual scenes. Further examples demonstrate spatial understanding and compositional generalization, while failures remain when salient regions omit important body parts or when reading and counting are required.

  • Captioning examples: The ResNet baseline hallucinates a toilet in a bathroom containing a couch but no toilet, whereas Up-Down correctly identifies the couch.The baseline error is attributed to language priors, while the Up-Down model handles the unusual scene composition.
  • Captioning examples: Attention captures fine details, including hands holding game controllers and sheep’s legs when generating “standing.”These examples illustrate that the approach avoids the trade-off between coarse and fine levels of detail.
  • Captioning examples: The generated captions suggest understanding of spatial relationships and successful captioning of a compositionally novel scene.The spatial example concerns generating the word “together.”
  • Captioning examples: A failure case mistakes a dog’s pose for laying rather than jumping, possibly because salient-region cropping misses the dog’s head and feet.The missed regions may prevent the model from recognizing the pose correctly.
  • VQA examples: The VQA model’s attention maps are often correctly focused despite limited reading and counting capabilities.The examples include both successful VQA results and failure cases.
Loading 1707.07998v3…