PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

Hyeonwoo Noh, Paul Hongsuck Seo, Bohyung Han

arXiv:1511.05756v1cs.CVcs.CLcs.LG

TL;DR

ImageQA must handle questions demanding different types and levels of image understanding, while constructing question-dependent classifiers with many parameters is challenging. The paper combines a CNN with a GRU-based parameter prediction network and hashing-based weight selection. It reports state-of-the-art performance on all three public ImageQA benchmarks.

  • Problem

    ImageQA involves heterogeneous recognition tasks, and predicting the large number of weights needed for a question-dependent fully connected layer is challenging.

  • Method

    A CNN uses a question-conditioned dynamic parameter layer whose weights are predicted by a GRU network, with hashing selecting candidate weights and end-to-end training.

  • Results

    The algorithm achieves state-of-the-art performance on DAQUAR, COCO-QA, and VQA.

  • Takeaways & Limitations

    The architecture handles multiple ImageQA recognition tasks with one CNN and achieves strong performance without more complex recognition processes such as object referencing.

Abstract

from arXiv · show

We tackle image question answering (ImageQA) problem by learning a convolutional neural network (CNN) with a dynamic parameter layer whose weights are determined adaptively based on questions. For the adaptive parameter prediction, we employ a separate parameter prediction network, which consists of gated recurrent unit (GRU) taking a question as its input and a fully-connected layer generating a set of candidate weights as its output. However, it is challenging to construct a parameter prediction network for a large number of parameters in the fully-connected dynamic parameter layer of the CNN. We reduce the complexity of this problem by incorporating a hashing technique, where the candidate weights given by the parameter prediction network are selected using a predefined hash function to determine individual weights in the dynamic parameter layer. The proposed network---joint network with the CNN for ImageQA and the parameter prediction network---is trained end-to-end through back-propagation, where its weights are initialized using a pre-trained CNN and GRU. The proposed algorithm illustrates the state-of-the-art performance on all available public ImageQA benchmarks.

1. Introduction

ImageQA unifies heterogeneous recognition problems by asking systems to answer questions about images, but different questions require different types and levels of understanding. The paper proposes question-dependent dynamic CNN parameters, uses hashing to control parameter growth, and reports state-of-the-art performance across three benchmarks.

  • Motivation: ImageQA targets holistic scene understanding by unifying diverse recognition problems in a question-answering task.Questions may concern different image properties and require different levels of understanding.
  • Related challenge: Existing deep approaches combine CNN image features with question descriptors and often treat answers as co-occurrences of joint features.These approaches use CNNs or bag-of-words representations for questions.
  • Approach: The proposed CNN adapts a dynamic parameter layer according to each question, allowing one architecture to address different recognition tasks.A GRU-based parameter prediction network generates candidate weights for the question-conditioned layer.
  • Approach: Hashing selects candidate weights for individual dynamic-layer parameters, reducing parameter count with little impact on network capacity.The technique is intended to make prediction of many dynamic weights more efficient.
  • Training: Fine-tuning a GRU pretrained on a large text corpus improves the network’s generalization performance.The motivation is the relatively small amount of language data in ImageQA datasets.
  • Results: The algorithm achieves state-of-the-art performance on DAQUAR, COCO-QA, and VQA, covering all three currently available benchmark datasets.The paper reports results across all public ImageQA benchmarks considered.

2. Related Work

Prior ImageQA methods include symbolic reasoning and deep learning approaches, while parameter prediction had mainly been studied in zero-shot learning. The paper positions dynamic parameter prediction with hashing as an unattempted way to handle ImageQA’s larger and more complex answer space.

  • ImageQA approaches: Earlier ImageQA work includes Bayesian reasoning with semantic segmentation and symbolic question reasoning, but predefined predicates limit representation of complex models.This approach differs from the predominantly deep-learning-based methods discussed in the section.
  • ImageQA approaches: Deep ImageQA systems commonly combine CNN image features with question representations, including joint embeddings or continuous bag-of-words features.The cited discussion notes that simple joint-feature classification can outperform LSTM-based embedding on COCO-QA.
  • Parameter prediction: Dynamic weight prediction had been explored for zero-shot classification, but ImageQA requires combining questions and answers across a substantially larger candidate space.The paper states that the earlier binary-classifier setting is not directly applicable to ImageQA.
  • Parameter prediction: Hashing had reduced parameters for fitting large neural networks in limited memory, but its use for dynamically predicting deep-network parameters had not been attempted.The paper presents this application as a gap in prior work.

3. Algorithm Overview

The proposed approach uses the question to define an adaptive recognition task within a CNN, replacing fixed classifier parameters with question-dependent weights. Hashing makes prediction of the large dynamic parameter set more tractable.

  • Algorithm Overview: The method adapts a CNN's dynamic parameter layer according to the input question instead of using one fixed recognition task.The dynamic parameters θd(q) are determined by question q, while static parameters remain fixed.
  • Algorithm Overview: DPPnet combines a CNN classification network with a question-driven parameter prediction network.The parameter prediction network uses the question to generate candidate weights for the dynamic layer.
  • Algorithm Overview: Hashing maps candidate weights to individual dynamic-layer weights, reducing the complexity of predicting a large weight matrix.A predefined hash function selects candidate weights for the dynamic parameter layer.

4. Network Architecture

The architecture pairs a VGG-based CNN with a GRU parameter predictor that generates question-dependent candidate weights. A hashing scheme shares those candidates across the dynamic layer to control parameter growth.

  • Network Architecture: The network contains a CNN classification network and a GRU-based parameter prediction network.The CNN includes a dynamic fully connected layer, while the predictor takes a question and generates candidate weights.
  • Classification Network: The VGG-based classification network removes its final layer and adds three fully connected layers, ending with an answer-classification layer.The classification layer has one output dimension per possible answer.
  • Classification Network: The dynamic layer is placed before the classification layer because it has fewer parameters and offers better scalability as answer classes grow.The fixed classification layer also provides a shared answer-space representation across recognition tasks.
  • Dynamic Parameter Layer: For input f_i, the dynamic layer computes f_o = Wd(q)f_i + b, where Wd(q) is constructed from the question.The dynamic weight matrix is parameterized as a function of q.
  • Parameter Prediction Network: The GRU encodes the question into h_T, and a fully connected layer transforms h_T into candidate weight vector p.GRU gates update hidden states directly, unlike LSTM's explicit memory cell.
  • Parameter Hashing: Hashing maps each dynamic-matrix location to an index in p, allowing one candidate parameter to be shared by multiple matrix elements.The independent sign hash ξ(m,n) helps remove bias from the hashed inner product.

5. Training Algorithm

The model is trained jointly by back-propagating answer error through both subnetworks. Training addresses limited language data, noisy CNN gradients, recurrent gradient behavior, and changing dynamic-layer activation distributions.

  • End-to-End Optimization: The classification and parameter prediction networks are jointly trained end-to-end by back-propagating answer error through both subnetworks.The derivatives are propagated by the chain rule and optimized with a first-order method.
  • Back-Propagation: Gradients for shared dynamic weights are accumulated across all connections that use the same predicted parameter.The indicator function identifies which dynamic-layer connections share each output of the parameter predictor.
  • Initialization: The method initializes the GRU from a skip-thought model trained on more than 74M sentences to transfer information from a large language corpus.This addresses the relatively limited language data available in ImageQA datasets.
  • CNN Fine-Tuning: CNN fine-tuning is modified because gradients below the dynamic parameter layer are noisy and straightforward fine-tuning typically fails to improve performance.The transferred VGG component is initially excluded while other network parameters are updated on the new datasets.
  • Implementation: Adam optimization uses learning rate 0.01, gradient clipping at 0.1, and validation-based stopping after five epochs without improvement.Gradient clipping handles gradient explosion from the GRU recurrent structure.
  • Training Stabilization: Batch normalization stabilizes changing dynamic-layer activation distributions, while GRU fine-tuning stops when overfitting begins.The network otherwise continues training after GRU fine-tuning is halted.

6. Experiments

The proposed network is evaluated across public ImageQA benchmarks using dataset-specific metrics and controlled experiments. It consistently outperforms existing approaches, while qualitative analyses show task adaptation alongside difficulty with occurrence-count questions.

  • Datasets: The evaluation covers VQA, COCO-QA, and DAQUAR, whose questions are paired with images and mostly single-word or short-phrase answers.DAQUAR provides reduced and all variants, while COCO-QA contains object, number, color, and location questions.
  • Evaluation Metrics: DAQUAR and COCO-QA use classification accuracy and WUPS, while VQA evaluates open-ended and multiple-choice answers by human-consensus accuracy.WUPS uses thresholded Wu-Palmer similarity with thresholds 0.9 and 0.0; VQA multiple-choice questions offer 18 candidate answers.
  • Controlled Experiments: Controlled VQA experiments compare DPPnet with CNN-FIXED, RAND-GRU, and CONCAT to assess dynamic prediction, GRU pre-training, and CNN fine-tuning.CONCAT combines CNN and GRU features without the proposed components, while RAND-GRU omits GRU pre-training and CNN fine-tuning.
  • Results: DPPnet achieves state-of-the-art performance on VQA and outperforms existing approaches consistently across COCO-QA, DAQUAR-reduced, and DAQUAR-all.The VQA comparison reports large margins, while the cross-benchmark result is stated without numerical values in the supplied passages.
  • Analysis: Fine-tuned GRU retrieval relies more on the tasks to be solved than on common subjective or objective words.The comparison is based on representative questions ordered by cosine similarity before and after GRU fine-tuning.
  • Qualitative Results: Qualitative results show adaptation to different question-dependent recognition tasks and effective answering of the same question across different images.The network often fails on occurrence-count questions because object detection is difficult to learn from image-level annotations.

7. Conclusion

The proposed ImageQA architecture combines a classification network with a question-conditioned parameter prediction network, using a dynamic parameter layer to adapt classification weights. Parameter hashing and weight sharing address the infeasible dimensionality of predicting every weight, while experiments report state-of-the-art performance on three datasets.

  • The dynamic parameter layer enables the classification network to adapt its weights according to the question.
  • The architecture uses two subnetworks: a classification network and a parameter prediction network.The classification network contains a dynamic parameter layer whose weights are adaptively determined through the parameter prediction network.
  • Parameter hashing and weight sharing make predicting the full dynamic-layer weight matrix feasible despite its large dimensionality.
  • The architecture achieves state-of-the-art performance on three different ImageQA datasets.The conclusion also states that this performance is achieved without more complex recognition processes such as object referencing.
Loading 1511.05756v1…