Source-linked AI summary
Pythia v0.1: the Winning Entry to the VQA Challenge 2018
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, Devi Parikh
TL;DR
Visual Question Answering models largely follow a shared design paradigm, motivating a modular research framework. Pythia v0.1 reimplements the up-down model and improves it through architectural, training, feature, augmentation, and ensembling changes, forming the basis of the winning VQA Challenge 2018 entry.
Problem
Most contemporary Visual Question Answering models follow a common paradigm of question encoding, image features, multimodal fusion, and answer classification.
Method
Pythia v0.1 starts from a modular up-down reimplementation and improves it through model, training, feature, augmentation, and diverse ensembling changes.
Results
The study reports significant performance improvements over the up-down baseline through its sequence of model and training changes.
Takeaways & Limitations
Pythia v0.1 provided the modular framework underlying FAIR’s winning entry to the VQA Challenge 2018.
Abstract
from arXiv · showhide
This document describes Pythia v0.1, the winning entry from Facebook AI Research (FAIR)'s A-STAR team to the VQA Challenge 2018. Our starting point is a modular re-implementation of the bottom-up top-down (up-down) model. We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2.0 dataset -- from 65.67% to 70.22%. Furthermore, by using a diverse ensemble of models trained with different features and on different datasets, we are able to significantly improve over the 'standard' way of ensembling (i.e. same model with different random seeds) by 1.31%. Overall, we achieve 72.27% on the test-std split of the VQA v2.0 dataset. Our code in its entirety (training, evaluation, data-augmentation, ensembling) and pre-trained models are publicly available at: https://github.com/facebookresearch/pythia
1. Introduction
Pythia v0.1 is a modular VQA research framework that formed the basis of FAIR’s A-STAR team’s winning VQA Challenge 2018 entry. It reimplements the bottom-up top-down model and improves it through subtle architectural changes.
- Pythia v0.1 is a modular framework for Visual Question Answering research and the basis of FAIR’s A-STAR team’s winning VQA Challenge 2018 entry.
- Most contemporary VQA models follow a modular design with question encoding, image feature extraction, multimodal fusion, attention, and answer classification.
- Pythia’s long-term goal is to support easy, modular research and development in VQA and related directions such as visual dialog.
- The system starts from a modular reimplementation of the bottom-up top-down model and improves its performance through a sequence of subtle but important changes.The paper summarizes these improvements in Table 1.
2. Bottom-Up and Top-Down Attention
The up-down model combines detector-based bottom-up visual attention with question-conditioned top-down attention and multimodal answer scoring. Pythia improves this framework through architectural changes, optimization adjustments, feature fine-tuning, additional data, and richer visual representations.
- Baseline up-down model: Bottom-up attention uses Faster RCNN with a ResNet-101 backbone pretrained on Visual Genome to extract object-region features.The model uses the entire Res-5 block as the detector’s second-stage region classifier.
- Baseline up-down model: Top-down attention uses the question to weight image objects, followed by Hadamard fusion and a sigmoid multi-label classifier for answer scores.The baseline ensemble reached 70.34% on the VQA 2.0 test-std split with 30 models trained using different seeds.
- Architectural changes: Pythia replaces gated tanh with weight normalization plus ReLU and uses element-wise multiplication instead of feature concatenation for top-down attention.Question representations use 300D GloVe embeddings, a GRU, and question attention; the best-performing multimodal hidden size was 5000.
- Feature fine-tuning: 68.49% test-dev accuracy is achieved by fine-tuning detector features with a fine-tuning learning rate set to 0.1 times the overall learning rate.The FPN-based detector exposes 2048D fc6 features and permits fine-tuning fc7 parameters with less storage, I/O, and computation than convolutional feature maps.
- Data augmentation: Additional Visual Genome and Visual Dialog data, plus mirrored VQA images with left-right token swaps, are used as data augmentation.For the added datasets, answers are replicated 10 times to match VQA’s evaluation format, and training stops at 22K iterations after a learning-rate reduction at 15K iterations.
- Visual representations: 70.01% test-dev and 70.24% test-std accuracy are obtained when using 100 object proposals for every image instead of an adaptive proposal count.The authors also combine grid-level ResNet152 features with bottom-up features to capture holistic spatial information and regions outside object proposals.