Source-linked AI summary
Learning to Compose Neural Networks for Question Answering
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein
TL;DR
Question answering needs to handle both images and structured knowledge bases while combining linguistic compositionality with continuous representations. The paper learns to assemble question-specific neural networks from composable modules using world-question-answer triples, achieving state-of-the-art results on two tasks. Its conclusions emphasize dynamic structure and continuous representations, while noting simplifying assumptions and difficulty modeling answer priors.
Problem
Question answering requires a model that can compositionally reason over both unstructured images and structured knowledge bases.
Method
The model jointly learns module parameters and a network structure predictor that assembles differentiable neural networks tailored to each question from world-question-answer triples.
Results
The dynamic neural module network achieves state-of-the-art results on two question answering tasks spanning natural images and United States geography.
Takeaways & Limitations
Dynamic structure tailors computation to each question, while continuous representations support differentiable reasoning over structured and complex continuous worlds.
Takeaways & Limitations
The layout-generation framework assumes relatively flat trees with at most one conjunction and one quantifier, and has difficulty modeling answer priors.
Abstract
from arXiv · showhide
We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains.
1 Introduction
The paper introduces a dynamic neural module network that learns to assemble composable modules into question-specific networks for images and structured knowledge bases, using only world-question-answer triples.
- The model translates questions into dynamically assembled neural networks and applies them to images or structured knowledge bases.
- It jointly trains a freely composable module collection and a network layout predictor that assembles complete networks for each question.
- The approach extends visual primitives to reason over structured world representations.
- Training uses only world-question-answer triples and requires no supervision of network layouts.
- The model achieves state-of-the-art performance on natural-image and United States geography question answering tasks.
2 Deep networks as functional programs
The model treats question answering as a functional program: learned structures compose neural modules that operate on attentions over images or entities in knowledge bases.
- A find module maps image features and a lexical item to an attention distribution over pixels.
- The same attentional operation can focus on entities in a knowledge base, representing a logical set-valued denotation with differentiable distributions.
- Other modules combine attentions or inspect them directly for operations analogous to conjunction, disjunction, and quantification.
- The system aims to automatically induce variable-free, tree-structured computation descriptors and uses them as network layouts.
- The dynamic neural module network learns to select layouts from automatically generated candidates.
3 Related work
Related work spans logical-form database QA, neural QA, and visually grounded attention models; this paper unifies perceptual and schema reasoning with differentiable execution.
- Database QA commonly maps strings to logical forms and evaluates them with a black-box execution model.
- The paper differs by providing a unified framework for perceptual and schema cases with continuous, differentiable evaluation results.
- Existing neural QA approaches include classification, shared question-answer embeddings, and attention over document words.
- Visual QA methods use convolutional image features, recurrent image-question models, or question-conditioned image attention.
- Earlier modular visual learning permitted limited per-question structural variation selected by deterministic syntactic processing.
4 Model
The model executes question-specific neural networks over world representations and learns to assemble those networks from candidate layouts without layout supervision.
- Network execution: The execution model assembles modules specified by layout z into a network that maps world representation w to an answer distribution.Intermediate module results flow to the root, whose output is denoted JzKw.
- Module types: Modules operate over Attention distributions or Labels distributions, supporting both visual regions and structured entities.Attention represents distributions over pixels or entities, while Labels represents distributions over answers.
- Parameterization: Module parameters are tied across instances of the same module, while lexical parameter arguments specialize behavior for particular words.Global weights are shared within a module type, whereas parameter-specific weights are associated with lexical arguments.
- Layout prediction: The layout model generates candidate structures from a fixed syntactic parse, then scores them using question and query-feature representations.The query features include counts of module types and their associated parameter arguments.
- Layout constraints: Candidate layouts are restricted to relatively flat trees with at most one conjunction and one quantifier, a simplifying assumption intended to cover most task examples.The construction associates dependency-parse fragments with modules and assembles them into full layouts.
- Learning: Hard layout selection is nondifferentiable, so the layout parameters are optimized with policy gradients using execution-phase negative log-probability as reward.The execution and layout parameters are learned jointly without layout supervision.
5 Experiments
The model is evaluated on visual VQA and structured geography QA, including a new quantifier-sensitive GeoQA variant. It achieves state-of-the-art VQA performance and outperforms logical, perceptual, and fixed-structure neural baselines on GeoQA.
- Datasets: The evaluation spans a large visual question answering dataset and a small, more structured geography-question dataset.The visual task uses more than 200,000 images, while GeoQA contains 263 examples.
- VQA: VQA experiments use the VQA 1.0 test-standard set, with VGGNet convolutional features from images resized to 448×448.Development data supports model selection and hyperparameter tuning, while final results come from the evaluation server.
- VQA: The VQA model uses simple candidate layouts containing only describe, and, and find modules, with at most two conjuncts.This configuration was found to perform best for the task.
- VQA results: The model achieves state-of-the-art VQA results, with dynamic networks providing a small gain most noticeably on “other” questions.It outperforms visual bag-of-words, fixed-structure dynamic-parameter, conventional attentional, and non-predictive neural module network baselines.
- GeoQA: GeoQA+Q distinguishes existential questions from set-denoting questions and expects Boolean answers for the existential form.The variant tests prediction of a wider variety of network structures than the original dataset.
- GeoQA results: 20% relative improvement over the fixed baseline is achieved on the quantifier dataset, while D-NMN also outperforms logical, perceptual, and fixed-structure models overall.Results use leave-one-environment-out cross-validation over 10 environments.
6 Conclusion
The dynamic neural module network combines differentiable module representations with question-specific network structures to answer queries across structured and unstructured worlds. The authors attribute its success to improved semantic expressiveness and dynamic computation, with applicability beyond question answering.
- The model answers queries about structured and unstructured information using only (question, world, answer) triples while learning network assembly and module weights.It assembles neural networks on the fly from an inventory of neural models and achieves state-of-the-art results on two tasks.
- Continuous neural predicate representations bypass combinatorial semantic-lexicon induction and can express reusable attributes and relations absent from a schema.These representations also extend compositional question-answering machinery to continuous worlds such as images.
- The approach extends compositional question answering across markedly different domains, including natural images and United States geography.The reported evaluation covers one visual task and one more compositional structured-world task.
- Dynamic network structures tailor computation to each question, using deeper networks for complex questions and representing combinatorially many queries with comparatively few parameters.The authors report gains in speed and sample efficiency even with very little training data.
- The authors expect these observations to transfer to instruction following, game playing, and language generation.