Source-linked AI summary
In Defense of Grid Features for Visual Question Answering
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, Xinlei Chen
TL;DR
The paper asks whether region features’ success in VQA depends on their format or instead on their semantic pretraining and spatial resolution. It revisits grid features, compares them across models and tasks, and finds comparable accuracy with substantially faster extraction, while enabling simpler end-to-end training. The authors report that grids can support strong VQA without region annotations in pretraining.
Problem
It is unclear whether region features outperform grid features because of region-specific advantages such as better localization and multiscale coverage.
Method
The paper compares grid and region features under matched detector pretraining, evaluates them across VQA models, benchmarks, and captioning, and explores end-to-end grid-based VQA.
Results
Grid features achieve on-par accuracy across VQA tasks and models and on captioning, while skipping region-related bottlenecks often yields speed-ups greater than an order of magnitude.
Takeaways & Limitations
Feature semantic content is more critical than region-versus-grid format, and effective grid representations can simplify and accelerate vision-language systems.
Takeaways & Limitations
Both region and grid features can fail even when attending to supporting concepts, and some questions may require specialized modules such as counting mechanisms.
Abstract
from arXiv · showhide
Popularized as 'bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA). However, it is not clear whether the advantages of regions (e.g. better localization) are the key reasons for the success of bottom-up attention. In this paper, we revisit grid features for VQA, and find they can work surprisingly well - running more than an order of magnitude faster with the same accuracy (e.g. if pre-trained in a similar fashion). Through extensive experiments, we verify that this observation holds true across different VQA models (reporting a state-of-the-art accuracy on VQA 2.0 test-std, 72.71), datasets, and generalizes well to other tasks like image captioning. As grid features make the model design and training process much simpler, this enables us to train them end-to-end and also use a more flexible network design. We learn VQA models end-to-end, from pixels directly to answers, and show that strong performance is achievable without using any region annotations in pre-training. We hope our findings help further improve the scientific understanding and the practical application of VQA. Code and features will be made available.
1. Introduction
The paper questions whether region features outperform grids because of localization or multiscale coverage, and finds grid features can match their accuracy when similarly pretrained. Across models, benchmarks, and tasks, grids also offer major speed and training-design advantages.
- Bottom-up attention represents images with detector-produced region features, which became widely used in VQA and related vision-language tasks.
- The paper asks whether regions are superior to grids because they localize objects better or capture coarse and fine-grained information through overlapping regions.
- Grid features from the same pretrained detector layer perform competitively with region features, while training modifications can make them comparable or sometimes better.
- The study attributes most of bottom-up attention’s effectiveness to Visual Genome object-and-attribute pretraining and high-resolution inputs, rather than region-versus-grid format.
- 0.02s versus 0.89s per image: grid features run 40+ times faster with slightly better accuracy using ResNet-50.
- Grid features support end-to-end VQA from pixels to answers without extra grounding annotations and permit more flexible model designs.
2. Related Work
Related work established region features as a dominant representation for vision-language benchmarks, while grid-based methods had already shown strength on diagnostic reasoning tasks. This paper revisits grids for general VQA and connects these research directions.
- Modern VQA models depend strongly on their underlying visual features, and many leading systems use region features directly.
- The paper shows that grids can be equally effective for general VQA and may bridge benchmark-focused and reasoning-focused lines of work.
- Region features are effective on VQA and COCO captions, while simple grid-based methods have shown strong performance on CLEVR reasoning benchmarks.
3. From Regions to Grids
The paper converts a bottom-up detector’s region extractor into grid-based alternatives while preserving comparable pretraining conditions. Direct C5 extraction already performs strongly, and 1×1 RoIPool improves the grid-oriented detector design.
- 3. From Regions to Grids: The approach initially constrains grid and region features to use the same pretraining task, while later end-to-end training removes that constraint.
- 3.1. Bottom-Up Attention with Regions: Bottom-up attention uses Faster R-CNN pretrained on Visual Genome object and attribute annotations with thousands of categories and hundreds of attributes.
- 3.1. Bottom-Up Attention with Regions: Region extraction requires proposal and postprocessing selection with NMS, followed by RoIPool and separate per-region network computation.
- 3.2. Grid Features from the Same Layer: Directly using shared C5 outputs converts the detector back toward a ResNet grid extractor because all final region features originate from C5.
- 3.3. 1×1 RoIPool for Improved Grid Features: 1×1 RoIPool represents each region with one vector, forcing each grid location to contain the information for its spatial region.
- 3.3. 1×1 RoIPool for Improved Grid Features: The table compares region and grid features on VQA 2.0 vqa-eval, including same-layer extraction and 1×1 RoIPool pretraining effects.
- 3.3. 1×1 RoIPool for Improved Grid Features: The modified detector uses shared ResNet features with fully connected region layers, altered strides and dilation during training, and an unchanged grid extractor during inference.
4. Main Comparison: Regions vs. Grids
Controlled comparisons show that grid features can match or exceed region features for VQA while avoiding region-specific computation. Performance depends strongly on pre-training and input resolution, whereas feature format has a smaller effect.
- Main comparison: 64.29 vs. 63.64 accuracy: C5 grid features from the detector perform competitively with widely used bottom-up region features.Both use the same detector, while ImageNet-only ResNet-50 features reach 60.76 accuracy.
- Main comparison: 0.73%: A 1×1 RoIPool-based variant improves grid-feature VQA accuracy and slightly surpasses the original region-based features.The same variant reduces VG object-detection average precision from 4.07 to 2.90 and lowers region-feature VQA accuracy to 63.94.
- Number of features: 200 regions: Region-feature accuracy saturates beyond this point, while grid features remain better even at 608 features.Grid feature count is determined by input size; a 600×1000 image produces 608 features on the 32-stride C5 map.
- Inference time: 48× faster: Grid-input VQA inference avoids region selection and feature computation, which account for 98.3% of region-model inference time.The comparison reports accuracy alongside inference time on VQA 2.0 test-dev, measured in milliseconds per image.
- Qualitative comparison: Both feature types attend to relevant image concepts, although region maps are more object-like and grid maps need not cover entire supporting concepts.The qualitative comparison associates successful answers with relevant concepts but not necessarily with precise object-shaped localization.
- Qualitative comparison: Both region and grid models still fail on some examples, where specialized modules such as counting components may be needed.These failures occur even when the models attend to supporting concepts.
5. Why do Our Grid Features Work?
The experiments identify input image size and pre-training task as major factors behind grid-feature performance, while attributes substantially improve VQA accuracy.
- Why do Our Grid Features Work?: Pre-training task and input image size are the two major differences examined between conventional ImageNet and proposed grid features.The comparison uses VQA 2.0 vqa-eval results and fixes attention on these two factors.
- Factor 1: Input Image Size: Larger input images improve grid-feature performance, with VG models continuing to benefit beyond 600×1000.ImageNet models saturate around 600×1000, whereas VG models continue improving at 800×1333.
- Factor 2: Pre-Training Task: Removing attribute supervision during VG pre-training drops VQA accuracy to the level of a pre-trained COCO model.This suggests attribute annotations are a major reason for VG models’ stronger performance.
- Factor 2: Pre-Training Task: Grid features consistently outperform region features across all tested attribute loss weights.The comparison is reported on the VQA 2.0 vqa-eval set.
6. Generalization of Grid Features
Grid features generalize across stronger backbones, VQA models, datasets, and image captioning while retaining substantial speed advantages over region features.
- Different backbone: 23.8× faster: grid features remain competitive with region features using a ResNeXt-101-32x8d backbone.The evaluation uses Faster R-CNN models trained on Visual Genome and VQA 2.0 test-dev.
- Different VQA model: +1.66 VQA improvement: grid features outperform regions with the MCAN model relative to reported MCAN results.The resulting single-model system achieves a state-of-the-art VQA 2.0 test-std result of 72.71.
- Different VQA task: 23× faster: grid features achieve comparable results to regions on the challenging VizWiz dataset.VizWiz contains cellphone images from visually-impaired users, with poor image quality, conversational questions, and unanswerable questions.
- Image captioning: Grid features achieve comparable image-captioning results to bottom-up region features on COCO.Evaluation uses BLEU4, METEOR, CIDEr, and SPICE without CIDEr optimization.
7. Towards End-to-end VQA
Grid features enable end-to-end VQA training from pixels to answers and support flexible architectures, while improving accuracy across several pre-training sources.
- Towards End-to-end VQA: 38 ms: grid-based Pythia runs substantially faster than region-based Pythia at 874 ms in the listed comparison.The table reports accuracy and inference time for Pythia, regions, and grids.
- Towards End-to-end VQA: 72 ms: grid-based MCAN is faster than region-based MCAN at 963 ms in the listed comparison.The corresponding accuracies are 72.59 for grids and 72.01 for regions.
- Training details: Joint training back-propagates answer-accuracy errors directly to convolutional grid feature maps.The first two residual blocks are fixed while the remainder is fine-tuned, and online features permit data augmentation.
- Results: End-to-end training boosts accuracy for VG-, ImageNet-, and YFCC-pre-trained grid models, with the largest improvements for ImageNet models.The models are evaluated on the VQA 2.0 test-dev set.
- Flexible network design: A Pyramid Pooling Module aggregates grid features across different spatial resolutions, expanding the architectural design space.The module is added on top of ResNet-50 grid features.
8. Conclusion
The paper concludes that grid features can match region features across vision-language tasks while removing costly region-processing bottlenecks. It emphasizes semantic content, pre-training, and end-to-end optimization over feature format alone.
- Conclusion: Grid features achieve on-par accuracy with region features across VQA tasks, VQA models, and image captioning.The conclusion presents grids as an alternative to widely used bottom-up region features.
- Conclusion: More than an order of magnitude: skipping region-related bottlenecks produces substantial speed-ups over region-based systems.The paper identifies semantic content as more critical than whether features use regions or grids.
- Conclusion: Strong VQA performance can come from VG object-and-attribute pre-training or end-to-end grid-feature training without region-level annotations.The conclusion notes that end-to-end training is easier with grids than with regions.
Appendix A. Details of Hyperparameters
The appendix records optimizer settings and learning-rate choices, including a reduced learning rate for image captioning and end-to-end Pythia training.
- Image captioning uses an initial learning rate of 0.002 instead of the default 0.01 because it produced slightly better results.
- End-to-end Pythia uses an initial learning rate of 0.002 and gradient clipping at 1 while fine-tuning ResNet feature extraction.
- SGD uses momentum 0.9 and weight decay 0.0001.
- Adamax uses β1=0.9 and β2=0.999 without weight decay.
- Adam for MCAN uses β1=0.9 and β2=0.98 without weight decay.
Appendix B. Region Features from FPN
The appendix compares FPN-derived region features with C4 region features and grid features, finding no clear FPN advantage for VQA and weaker results than C4 and grids in MCAN.
- FPN uses an entire ResNet model to compute region features, unlike the original Faster R-CNN C4 model used in bottom-up attention.
- FPN region features show no clear VQA advantage over the original C4 model, which performs on par with grid features.
- With MCAN, FPN performs better than the result reported in [50] but worse than both C4 and the authors’ grid features.
- The comparison spans different backbones, VQA models, VQA tasks, and image captioning.
Appendix C. Details of PPM
The appendix details a Pyramid Pooling Module that augments grid convolution features with multiscale pooled representations before VQA prediction, alongside broader speed and generalization comparisons.
- PPM architecture: PPM applies adaptive average pooling to a ResNet grid feature map at 1×1, 4×4, and 8×8 spatial resolutions.
- PPM architecture: Three convolution layers with batch normalization and ReLU process the pooled features, using kernel size 1 and output dimension 512.
- Evaluation context: Table 9 compares VQA accuracy and inference-time breakdowns for grid, C4 region, and FPN region features on VQA 2.0 test-dev.
- Evaluation context: Table 10 evaluates generalization across backbone, VQA model, VizWiz, and COCO image captioning settings.
- PPM architecture: The original grid feature map is concatenated with the three PPM outputs as input to the VQA model.