PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Module Networks

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein

arXiv:1511.02799v4cs.CVcs.CLcs.LGcs.NE

TL;DR

Visual question answering requires combining image understanding with the compositional structure of natural-language questions, but existing approaches often use monolithic representations. The paper introduces neural module networks that parse each question, dynamically assemble reusable neural modules, and jointly train the resulting networks; it reports state-of-the-art performance on existing and newly introduced visual QA tasks.

  • Problem

    Visual question answering requires answering natural-language questions about images while handling questions with compositional structure.

  • Method

    The method parses questions into structured queries and dynamically assembles reusable, heterogeneous neural modules, combining their output with an LSTM question encoder.

  • Results

    The approach achieves state-of-the-art visual question-answering performance, especially on object- and attribute-based questions and highly compositional shape questions.

  • Takeaways & Limitations

    Neural module networks provide a general-purpose framework for dynamically assembling jointly trained neural components into deep networks for complex reasoning tasks.

  • Takeaways & Limitations

    Network-structure prediction and parameter learning remain strictly separated, although the paper suggests they could be solved jointly.

Abstract

from arXiv · show

Visual question answering is fundamentally compositional in nature---a question like "where is the dog?" shares substructure with questions like "what color is the dog?" and "where is the cat?" This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning *neural module networks*, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

1. Introduction

The paper introduces neural module networks, which use linguistic structure to dynamically compose jointly trained neural modules for visual question answering. It combines neural representations with compositional question analysis and evaluates the approach on natural-image and synthetic visual QA tasks.

  • Problem: Visual question answering requires predicting answers from images and natural-language questions, demanding understanding of both visual scenes and language.Existing approaches commonly represent questions as bags of words or encode them with recurrent networks before classification.
  • Method: The model uses a natural-language parser to lay out reusable modules, with an additional sequence model providing sentence context for visual QA.Figure 1 illustrates attention focused on an object passing to a location classifier, with messages including image features, attentions, and classification decisions.
  • Contribution: Neural module networks dynamically compose jointly trained neural modules into deep networks according to a question’s linguistic structure.The architecture uses a parser to determine computational units and module relationships, while remaining in the domain of visual features and attentions.
  • Contribution: The approach integrates neural-network representational capacity with the compositional structure associated with symbolic semantic approaches, without using logical truth-value reasoning.The resulting networks are assembled from specialized modules that are jointly learned.
  • Evaluation: The evaluation covers established VQA datasets and a new synthetic dataset designed for more complex, highly compositional questions.On VQA, the method performs comparably to or better than existing approaches and specifically outperforms prior work on compositional questions.
  • Scope: The architecture may also apply beyond visual question answering, including visual referring-expression resolution and question answering over natural-language texts.

2. Motivations

The paper motivates modular networks by observing that different tasks need different network topologies while sharing useful intermediate representations. It therefore treats question answering as many language-specified tasks and predicts a reusable computation structure for each question.

  • Motivation: Different computer-vision tasks require different deep-network topologies, so no single network is appropriate for every task.
  • Motivation: Networks are empirically modular because intermediate representations learned for one task can benefit many others.
  • Question answering: Question answering can be viewed as highly multitask learning in which each instance specifies a novel task only noisily through language.Simple and complex questions may require retrieving different numbers and kinds of facts from an image.
  • Framework: The proposed framework predicts each question’s computation structure and constructs an appropriately shaped network from reusable modules learned jointly.Modules specialize to tasks such as identifying properties and spatial relations while remaining part of a shared inventory.

3. Related work

Related work spans visual question answering, compositional semantics, and structurally related neural architectures. The paper distinguishes its approach by dynamically assembling heterogeneous modules and jointly training them while operating over visual features and attentions.

  • Visual question answering: Visual QA methods include logical systems, neural sequence models, and attention-based grounding approaches for connecting questions to images.The evaluated VQA dataset is larger and more visually varied than earlier alternatives discussed in the paper.
  • Visual question answering: Classical QA approaches use semantic parsers but rely on fixed logical inference, whereas neural QA models map joint image-text embeddings directly to answer distributions.
  • Compositional semantics: Compositional-semantics research learns to answer questions over structured knowledge representations, while instruction-following models impose discrete planning over continuous control.
  • Neural architectures: Recurrent and recursive neural networks select input-dependent structures, and some memory networks resemble fixed graphs of attention modules followed by classification.
  • Neural module networks: The paper’s contribution is dynamically assembling graphs whose modules perform heterogeneous computations and pass raw features, attentions, or classification predictions.The authors state they are unaware of prior work jointly training such mixed collections of modules.

4. Neural module networks for visual QA

The visual QA system parses each question into a symbolic structure, maps that structure to a dynamically assembled neural module network, and combines its output with an LSTM question encoder. Reusable attention, transformation, combination, classification, and measurement modules operate on continuous visual representations and are jointly trained.

  • Model formulation: Each training example consists of a natural-language question, an image, and an answer, while the model predicts a distribution p(y | w, x; θ) using a layout predictor and module collection.The layout predictor maps strings to networks, which process the image and possibly the question before producing label probabilities.
  • Modules: The module inventory operates on images, unnormalized attentions, and labels, with attention space carrying most of the task’s compositional phenomena.The paper characterizes the contribution as identifying composable visual primitives, while noting that additional data types could be added later.
  • Modules: Attention modules produce heatmaps for concepts, re-attention modules transform attentions, combination modules merge attentions, and classification modules map attended image regions to label distributions.Measurement modules instead map an attention alone to labels, supporting object-existence judgments and counting because attentions are unnormalized.
  • From strings to networks: A semantic parser converts questions into dependency-based symbolic queries, and the resulting structure determines module identities and network connections.Leaves become attend modules, internal nodes become re-attend or combine modules, and roots become measure modules for yes/no questions or classify modules otherwise.
  • Broader applicability: The learned modules can be assembled by outside users for visual SQL-like queries without relying on natural language specifically.The paper presents the architecture as potentially applicable beyond visual QA, including referring-expression resolution and text question answering.
  • Answering natural language questions: The final model combines the neural module network with an LSTM encoder to retain grammatical and semantic regularities discarded by aggressive question simplification.The two answer distributions are geometrically averaged with weights dynamically reweighted using text and image features.

5. Training neural module networks

Training maximizes the likelihood of the observed answers through the dynamically structured networks. Adaptive per-weight learning rates are used, and module behaviors emerge from end-to-end joint training rather than manual specification.

  • Objective: The training objective maximizes the likelihood of the data, with every assembled network producing a probability distribution over labels.The final module in each network is designed to output that distribution.
  • Optimization: AdaDelta is used because dynamically structured networks update some weights more frequently than others, making adaptive per-weight learning rates outperform simple gradient descent.The experiments use AdaDelta without hyperparameter search over step sizes.
  • Joint learning: Module labels such as cat and and are not manually assigned behaviors; the corresponding modules acquire their functions through end-to-end training.The paper explicitly notes that detect[cat] and combine[and] are neither fixed nor initialized as particular recognizers or attention operations.

6. Experiments: compositionality

The paper introduces SHAPES to test deep semantic compositionality under complex questions and distractors, then evaluates whether neural module networks can learn and generalize these structures.

  • Motivation: Existing natural-image VQA datasets mostly use simple questions requiring one or two extracted facts, with limited testing against distractors.This motivates a synthetic evaluation focused on compositional reasoning.
  • Dataset: SHAPES contains complex questions about colored-shape arrangements, combining two to four attributes, object types, or relationships across 15,616 images and 244 questions.All questions have yes-or-no answers, preventing mode guessing as a viable strategy.
  • Dataset: Good SHAPES performance requires recognizing shapes and colors while understanding spatial and logical relations among object sets.The dataset is designed to place compositional phenomena at the forefront.
  • Scope: SHAPES is intended as a necessary but insufficient condition for robust visual question answering, so it should be used alongside natural-image datasets.The authors explicitly limit what success on this synthetic benchmark establishes.
  • Results: The model achieves high SHAPES accuracy and outperforms the previous baseline, especially on highly compositional questions.The table defines size as the number of modules needed to instantiate the appropriate neural module network.
  • Results: Removing size-6 questions from training does not reduce performance and may slightly improve it, indicating generalization to more complex unseen question structures.Using linguistic information, the model extrapolates learned visual patterns to harder questions.

7. Experiments: natural images

The paper evaluates neural module networks on the human-annotated VQA dataset, finding stronger performance than prior approaches while exposing weaknesses in binary questions and parsing complex queries.

  • Dataset: The VQA dataset contains more than 200,000 images, each paired with three questions and ten answers per question, generated by human annotators.The model is trained using the standard train/test split.
  • Results: The full model outperforms previous VQA approaches, scoring particularly well on questions not involving a binary decision.NMN+LSTM is the full model, while NMN removes the whole-question LSTM.
  • Results: Performance is especially strong for object, attribute, and number answers, but worse than a sequence baseline for yes/no questions.Training-set accuracy suggests that overfitting contributes to the yes/no weakness.
  • Future work: A sequence-only ensemble might improve results, while redesigning the measure module is proposed to reduce overfitting effects.This is presented as future work within the neural module network framework.
  • Limitation: Parser inspection suggests that 80–90% of simple object-property questions are correctly analyzed, whereas more complicated questions more often acquire irrelevant predicates.The authors suggest joint learning as one possible way to fix these parser errors.
  • Error analysis: Observed errors include plausible semantic confusions, lexical variation, and answers that are plausible but unrelated to the image.Examples include interpreting cardboard as leather and locating a horse in a pen rather than a barn.

8. Conclusions and future work

The paper presents neural module networks as a general framework for dynamically assembling jointly trained neural modules, achieving strong visual question answering performance while pointing toward broader neural programs.

  • Neural module networks dynamically assemble collections of jointly trained neural modules into arbitrary deep networks.
  • The approach achieves state-of-the-art performance on existing visual question answering datasets, especially for questions answered by an object or an attribute.
  • On a new dataset of highly compositional questions about simple shape arrangements, the approach substantially outperforms previous work.
  • The paper leaves joint learning of network structures and parameters as future work, having maintained a strict separation between them.
  • The experiments suggest a broader paradigm in which neural-network parts are composed into programs for complex reasoning tasks beyond visual question answering.
Loading 1511.02799v4…