PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering

Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, Dacheng Tao

arXiv:1708.03619v2cs.CV

TL;DR

VQA requires fine-grained understanding of images and questions, expressive multimodal fusion, and prediction over correlated alternative answers. The paper integrates co-attention, MFH pooling, and KL-divergence loss into a unified model, with an ensemble reaching 68.02% on test-dev and 68.16% on test-challenge and ranking second in VQA Challenge 2017.

  • Problem

    VQA must jointly represent image and question content, capture their complex interactions, and model correlations among multiple diverse answers.

  • Method

    The paper combines co-attention for image and question representations, MFH for multimodal fusion, and KL divergence for answer prediction in a unified deep network.

  • Results

    68.02% on the test-dev set and 68.16% on the test-challenge set were achieved by a nine-model ensemble, ranking second in VQA Challenge 2017.

  • Takeaways & Limitations

    The proposed MFH models achieve state-of-the-art performance on real-world VQA datasets, while KL divergence provides faster convergence and better answer-prediction performance than state-of-the-art strategies.

Abstract

from arXiv · show

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multi-modal feature fusion that is able to capture the complex interactions between multi-modal features; 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a `co-attention' mechanism is developed by using a deep neural network architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multi-modal feature fusion, a generalized Multi-modal Factorized High-order pooling approach (MFH) is developed to achieve more effective fusion of multi-modal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the KL (Kullback-Leibler) divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A deep neural network architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA datasets and win the runner-up in VQA Challenge 2017.

I. INTRODUCTION

VQA requires fine-grained image and question understanding, expressive multimodal fusion, and answer prediction that accommodates correlated alternative answers. The paper addresses these needs with co-attention, MFH pooling, and KL-divergence-based learning within a unified architecture.

  • VQA requires fine-grained understanding of image and question semantics, complex reasoning, and accurate best-matching answer prediction.
  • The task involves extracting discriminative representations, fusing visual and textual features, and learning a classifier for answer prediction.
  • Co-attention jointly learns image and question attentions to reduce irrelevant features and obtain more discriminative representations.
  • MFH cascades multiple MFB blocks to capture more complex multimodal correlations and improve image-question representations and VQA performance.
  • KL divergence models consistency among diverse annotated answers, enabling faster convergence and slightly better answer-prediction accuracy.
  • An ensemble of the proposed models achieves state-of-the-art performance on real-world VQA datasets, while normalization techniques are reported as extremely important in bilinear pooling models.

II. RELATED WORK

Prior VQA research includes coarse joint embeddings, attention-based fine-grained embeddings, and external-knowledge models. The paper emphasizes limitations in global-feature noise, multimodal fusion expressiveness, and handling complex reasoning or common-sense knowledge.

  • Existing VQA approaches are grouped into coarse joint-embedding, fine-grained attention-based joint-embedding, and external-knowledge models.
  • Coarse joint-embedding models combine global image and question features, often using concatenation and a linear classifier for answer prediction.
  • Global features can contain irrelevant information, motivating visual attention methods that adaptively select local image features for each question.
  • External knowledge benefits questions requiring complex reasoning or common sense, but existing approaches are limited to specific datasets or ineffective on benchmarks.

B. Multi-modal Bilinear Models for VQA

The section motivates bilinear fusion as a way to capture multimodal feature interactions beyond simple linear combinations, then introduces factorized bilinear pooling as a more efficient alternative.

  • Linear fusion by concatenation or element-wise summation may inadequately represent interactions between visual and textual features with different distributions.
  • Bilinear pooling captures pairwise interactions between feature dimensions but requires many parameters, increasing computational cost and over-fitting risk.
  • MFB factorizes bilinear projections into low-rank matrices, combines projected modalities through element-wise multiplication, and applies sum pooling to produce compact outputs.
  • MFB consists of an expansion stage followed by a squeeze stage using dropout, sum pooling, and normalization.
  • MFB is more expressive than MLB at the same output dimensionality because it expands features before multiplication and then compresses them through pooling and normalization.

B. From Bilinear Pooling to Generalized High-order Pooling

The paper generalizes bilinear pooling by cascading multiple MFB blocks, producing higher-order multimodal representations whose size and output dimensionality scale with the number of blocks.

  • MFH extends second-order bilinear pooling to generalized high-order pooling to capture more complex interactions among multimodal feature dimensions.
  • Each MFB block computes an internal expanded feature and transforms it into an output feature through sum pooling and normalization.
  • MFH modifies the MFB expansion stage so multiple MFB blocks can be cascaded.
  • The MFH output concatenates the outputs of p MFB blocks, yielding a feature in R^op.
  • MFH model size and output dimensionality grow linearly with p, and experiments use p < 4 to control complexity and training time.

IV. NETWORK ARCHITECTURES FOR VQA

The VQA architectures extract image and question representations, fuse them with MFB or MFH, and classify the fused representation; a second architecture adds co-attention learning.

  • The model integrates image and question representations with MFB or MFH, treating each answer as a class for multi-class prediction.
  • The MFH design is organized around cascading p MFB blocks, with MFB as the special case p = 1.
  • The baseline architecture uses one MFB or MFH module for ablation studies and comparisons with other approaches.
  • The co-attention architecture jointly characterizes fine-grained correlations between multimodal features to improve representation capability.

A. The Baseline Model

The baseline represents images with pretrained ResNet features and questions with an LSTM, then fuses both modalities and predicts an answer through an N-way classifier.

  • Images are resized to 448 × 448, and 2048-D pool5 features with ℓ2 normalization are extracted from a 152-layer ImageNet-pretrained ResNet.
  • Questions are tokenized, converted to one-hot vectors with maximum length T, embedded, and processed by an LSTM with 1024 hidden units.
  • MFB or MFH fuses the image and question features into z, which an N-way classifier uses to predict the best-matching answer.
  • All weights except those of ResNet are optimized jointly in an end-to-end manner.

B. The Co-Attention Model

The co-attention model jointly learns question and image attention over fine-grained features, while modeling answer distributions to account for correlated annotations.

  • Co-attention architecture: Image attention identifies question-relevant spatial regions instead of relying on the entire image.Different questions can require different image regions, making question-conditioned image attention useful for answer prediction.
  • Co-attention architecture: Question and image attention modules use sequential convolutional, ReLU, and softmax layers to produce weighted attentive features.Multiple attention maps can be generated and concatenated to increase representation capacity.
  • Co-attention architecture: Question attention is self-attentive, whereas image attention uses both image and question features.The design assumes that question keywords can be inferred without viewing the image.
  • Answer correlation modeling: The model addresses answer diversity by representing multiple annotations as a distribution over possible answers.This captures correlations between alternative expressions such as “a little dog” and “a puppy.”
  • Answer correlation modeling: KL-divergence loss directly trains predictions against the answer distribution, avoiding sampling uncertainty and potentially reducing convergence time.KL divergence differs from multi-label cross-entropy by an additional constant term, so the two are equivalent during optimization.

VI. EXPERIMENTS

The experiments evaluate the proposed fusion and attention-related approaches on VQA datasets using standard dataset splits and answer-vote accuracy.

  • Experimental design: Experiments evaluate MFB and MFH against existing state-of-the-art approaches and analyze why the proposed models are effective.The study includes ablations and comparisons across multiple VQA datasets.
  • Experimental design: The co-attention architecture combines fine-grained image and question features with jointly modeled question and image attention.It supports either MFB or MFH for multimodal fusion.
  • Datasets and evaluation: VQA-1.0 contains approximately 200,000 MS-COCO images, three questions per image, and ten answers per question.The dataset provides Open-Ended and Multiple-Choices evaluation tasks, with train, validation, test, and test-dev splits.
  • Datasets and evaluation: VQA accuracy is computed from the number of annotator votes assigned to the predicted answer.The evaluation therefore incorporates agreement among multiple annotators.

2) VQA-2.0:

Experiments compare multimodal fusion models, normalization settings, and high-order pooling variants using VQA evaluation data and training curves.

  • VQA-2.0: VQA-2.0 adds training samples and balances questions by pairing each question with two images producing different answers.Results are reported for the Open-Ended validation task using models trained on the training set.
  • Fusion-model ablations: The ablation study evaluates MFB and MFH efficacy, including comparisons with feature-combination baselines and bilinear models.Table I uses the same input features without external datasets or attention mechanisms.
  • Fusion-model ablations: MFB(k=5,o=200) exceeds EltwiseProd by 2.1 points, while MFB(k=5,o=1000) exceeds EltwiseProd+FC+ReLU by 2.2 points.These comparisons support second-order bilinear pooling over first-order fusion models under similar model sizes.
  • Fusion-model ablations: MFB(k=5,o=1000) improves by about 1.0 points over MCB and MLB, while MFB(k=5,o=200) achieves similar results to MCB with 1/3 the parameters.The smaller model also uses 2/3 of MCB’s GPU memory, supporting training with larger batch sizes.
  • Training behavior: MFB’s validation accuracy is relatively robust, whereas MCB performance gradually falls after 25,000 iterations, indicating overfitting for MCB’s high-dimensional output.The comparison is shown in training and validation curves.
  • Normalization: ℓ2 normalization improves performance by about 3 points, while power normalization improves it by 0.5 points.Normalization restricts neuron values and produces more stable output distributions during training.
  • High-order pooling: MFH2 and MFH3 outperform MFB by about 0.7 points on test-dev, but MFH3 is slightly worse than MFH2.The authors attribute this to saturated representation capacity at p=2 and use MFH2 thereafter.

2) Answer Correlation Modeling Strategies:

The experiments compare answer-label strategies and find that modeling answer correlations improves VQA performance, with KLD converging faster than answer sampling.

  • Answer correlation strategies: Answer Sampling and KLD significantly outperform Max Prob on VQA-1.0 by modeling correlations among multiple answers.Max Prob uses the most frequent answer as a single label and does not model answer correlation.
  • Answer correlation strategies: KLD converges faster and achieves slightly better accuracy than Answer Sampling, especially for the more complex MFB+CoAtt model.The comparison uses validation accuracies for MFB and MFB+CoAtt.

D. Results on the VQA-1.0 Dataset

On VQA-1.0, MFB improves multimodal fusion over prior bilinear approaches, while co-attention and MFH provide further gains. Ensembles of these models achieve new state-of-the-art results, though attention errors still reveal unresolved weaknesses.

  • Single-model comparisons: MFB significantly outperforms existing comparative approaches on both open-ended and multiple-choice VQA-1.0 tasks.With attention, MFB+Att consistently outperforms the next-best MCB+Att model.
  • Single-model comparisons: Co-attention improves MFB+Att by 0.5 overall-accuracy points on the open-ended task and significantly improves each question type.The added self-attention module models question features in addition to image attention.
  • MFH and external data: Replacing MFB with MFH steadily improves the authors’ models by about 0.7∼1.1 points.A single MFH+CoAtt+GloVe model surpasses the best published ensemble of seven MLB or MFB models on the test-standard set.
  • MFH and external data: Adding pretrained GloVe and Visual Genome further improves performance, with MFH+CoAtt+GloVe+VG significantly outperforming the best reported single model on both tasks.
  • Ensemble results: The ensemble of MFH models achieves a further 0.8-point improvement over the MFB ensemble and establishes a new VQA-1.0 state-of-the-art.The MFB ensemble already exceeds the next-best result by 1.5 points on OE and 2.2 points on MC.
  • Attention analysis: Attention commonly focuses on key words and relevant image regions, but neglected question words and unrecognized or misclassified visual content still cause errors.The reported failures include missed words such as “catcher” or “bottom,” unrecognized flags, and misclassified meat in counting questions.

E. Results on the VQA-2.0 Dataset

On VQA-2.0, single MFB and MFH models with co-attention and GloVe surpass the baselines, while ensembles achieve high leaderboard performance. Using Faster R-CNN visual features further raises the reported accuracy.

  • Single-model results: Single MFB and MFH models with CoAtt+GloVe significantly surpass all baseline approaches on VQA-2.0.These models do not use Visual Genome data augmentation in this comparison.
  • Single-model results: MFB and MFH are about 2.7 and 3.5 points higher than MCB, respectively, when test-dev and test-standard differences are neglected.
  • Ensemble results: A nine-model ensemble reaches 68.02% on test-dev and 68.16% on test-challenge, ranking second place, tied with another team, in VQA Challenge 2017.
  • Visual features and leaderboard: With Faster R-CNN features pretrained on Visual Genome, an eight-model MFH ensemble reports 70.92% on VQA-2.0 test-standard, versus 70.34% for the champion team’s up-to-30-model result.The authors state that this places their model first on the real-time leaderboard as of 15 March 2018.
  • Model design: The overall architecture combines co-attention, MFH multimodal fusion, and KL-divergence answer prediction within a unified VQA model.The paper reports state-of-the-art or comparable performance on two large-scale real-world VQA datasets.
Loading 1708.03619v2…