Source-linked AI summary
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Kate Saenko
TL;DR
VQA requires compositional reasoning, but prior modular systems depend on brittle parsers and restricted layouts. N2NMNs learn question-specific network structures and module parameters end to end, achieving strong CLEVR results while producing interpretable architectures.
Problem
Prior NMN implementations rely on external parsers and parser-limited module configurations rather than learning suitable structures from data.
Method
N2NMNs use a sequence-to-sequence RNN layout policy to predict question-specific modular network layouts, with soft attention providing module parameters.
Results
N2NMNs achieve state-of-the-art results on CLEVR by a wide margin and improve further when end-to-end optimization follows behavioral cloning from expert layouts.
Takeaways & Limitations
The model can handle complicated reasoning problems while discovering interpretable network architectures specialized for individual questions.
Takeaways & Limitations
The original NMN question parser does not work on CLEVR, so the NMN comparison uses the same expert layout as N2NMNs.
Abstract
from arXiv · showhide
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems. For example, to answer "is there an equal number of balls and boxes?" we can look for balls, look for boxes, count them, and compare the results. The recently proposed Neural Module Network (NMN) architecture implements this approach to question answering by parsing questions into linguistic substructures and assembling question-specific deep networks from smaller modules that each solve one subtask. However, existing NMN implementations rely on brittle off-the-shelf parsers, and are restricted to the module configurations proposed by these parsers rather than learning them from data. In this paper, we propose End-to-End Module Networks (N2NMNs), which learn to reason by directly predicting instance-specific network layouts without the aid of a parser. Our model learns to generate network structures (by imitating expert demonstrations) while simultaneously learning network parameters (using the downstream task loss). Experimental results on the new CLEVR dataset targeted at compositional question answering show that N2NMNs achieve an error reduction of nearly 50% relative to state-of-the-art attentional approaches, while discovering interpretable network architectures specialized for each question.
1. Introduction
N2NMNs address the limitations of fixed, parser-dependent VQA systems by learning question-specific modular network layouts and textual module parameters directly from data.
- Motivation: VQA often requires compositional reasoning over multiple objects, properties, and comparisons rather than only joint image-text comprehension.The paper motivates this with questions requiring object localization, property inspection, and comparison.
- Motivation: Conventional VQA models show strong performance, but limited evidence links those gains to explicit compositional reasoning.The paper notes that performance can instead reflect statistical biases in question-answer data.
- Contribution: N2NMNs learn to parse language and compose an appropriate modular network architecture directly from textual input for each question.The model predicts instance-specific structures and applies them to images.
- Limitations of prior work: Previous modular networks depend on external parsers whose handcrafted adaptations can fail to produce valid layouts for language-and-vision tasks.Existing approaches either rely on parser-generated layouts or use fixed structures, limiting flexibility across questions.
- Contribution: The model learns a dynamic layout policy and soft-attention module parameterization without external linguistic resources at test time.This replaces hard-coded word assignments with attention over question words.
- Results: N2NMNs achieve state-of-the-art performance on CLEVR by a wide margin while predicting expert-provided layouts with near-perfect accuracy.The paper reports both layout-prediction accuracy and broad benchmark performance as evidence of the approach’s effectiveness.
2. Related work
Prior modular VQA methods construct dynamic networks but generally depend on external parsers or fixed layouts, motivating models that learn question-specific structures end to end.
- Neural module networks: Neural Module Networks construct instance-specific deep networks from reusable modules according to an input-dependent layout.Modules can be jointly trained across multiple structures to support reusable compositional behaviors.
- Related modular approaches: Compositional modular networks avoid parsing but restrict every referring expression to a fixed subject-relationship-object layout.That architecture does not search over alternative structures.
- Learning network architectures: Architecture-search research explores evolutionary algorithms, Bayesian methods, and reinforcement learning for discovering neural network structures from data.The paper identifies reinforcement-learning controllers as the closest general architecture-search precedent.
- Reasoning evaluation: VQA performance improvements do not necessarily demonstrate stronger reasoning because models can exploit question-answer statistics or bag-of-words representations.These findings motivate benchmarks and methods that more directly test compositional reasoning.
- Question-specific inference: Most prior VQA systems use one fixed inference structure for every question, despite potentially differing optimal reasoning procedures.This motivates inference structures specialized to individual inputs.
3. End-to-End Module Networks
N2NMNs dynamically assemble question-specific module networks from layouts predicted directly from text. The model jointly learns module parameters, textual attention, layout prediction, and end-to-end training with expert-policy initialization.
- Model overview: N2NMNs combine co-attentive neural modules with a layout policy that predicts a question-specific structure for dynamically assembling a network.Modules solve subtasks, while the layout policy selects their computational arrangement.
- Model overview: A predicted functional expression such as count(relocate(find())) is parameterized using text vectors before the corresponding network is assembled.Function applications can receive parameter vectors predicted from question text.
- Attentional neural modules: Modules consume attention maps and visual or textual features, producing attention maps or answer-score vectors for downstream reasoning.Find localizes objects or attributes, relocate transforms attention, and logical operators combine attention maps.
- Attentional neural modules: Unlike hard-coded textual assignments, each module obtains question information through soft attention over the question words.The attended word embeddings form a textual feature for each module.
- Layout policy with sequence-to-sequence RNN: The layout policy searches over all possible module expressions, linearizes them into module-token sequences, and predicts them with an attentional sequence-to-sequence RNN.At test time, beam search selects a maximum-probability layout before network assembly.
- End-to-end training: Training jointly optimizes the layout policy and neural modules by minimizing expected question-answering loss, using reinforcement learning for discrete layouts.Behavioral cloning from expert layouts supplies supervised initialization during training but is not used at test time.
4. Experiments
Experiments evaluate N2NMNs on SHAPES and CLEVR using expert-guided cloning, policy search, and end-to-end reinforcement learning. The model predicts interpretable layouts and improves over prior modular and non-modular approaches on complex reasoning tasks.
- 4.1. Analysis on the SHAPES dataset: SHAPES contains 15,616 image-question pairs with 244 questions on 3 × 3 grids, requiring compositional reasoning about shapes and spatial relations.Ground-truth parses provide expert layout supervision.
- 4.1. Analysis on the SHAPES dataset: 100% accuracy is achieved on SHAPES with behavioral cloning from the expert layout policy.Policy search from scratch also performs well without expert supervision.
- 4.2. Evaluation on the CLEVR dataset: CLEVR emphasizes long, complex inference chains over photorealistic scenes containing objects with varied shapes, colors, materials, sizes, and occlusions.Experiments use VGG-16 features augmented with spatial coordinates.
- Method and training: N2NMNs use a layout policy to assemble question-specific networks from co-attentive modules, with reinforcement learning jointly optimizing layouts and module parameters.The policy searches a larger layout space than parser-restricted approaches and can exploit downstream computation.
- 4.2. Evaluation on the CLEVR dataset: Without expert demonstrations, policy optimization from scratch surpasses most prior work, while behavioral cloning outperforms previous approaches in overall CLEVR accuracy.The model also outperforms an NMN baseline trained with the same expert layout.
- 4.2. Evaluation on the CLEVR dataset: Policy search after cloning consistently improves accuracy across CLEVR question types, with especially large gains on compare color questions.The results suggest the manually constructed expert policy is not always optimal.
5. Evaluation on the VQA dataset
On real-image VQA, N2NMNs use a syntactic parser only during training to construct expert layouts. They outperform prior modular methods and slightly exceed MCB while retaining an explicitly visible reasoning procedure.
- Training setup: The VQA experiment uses syntactic parses to construct expert layouts during training, while avoiding the parser at test time.Different visual features are used for fair comparison with other methods.
- Results: N2NMNs significantly outperform NMN and D-NMN using the same LRCN VGG-16 image features.This compares the methods under matched modular-model visual features.
- Results: N2NMNs achieve slightly higher performance than MCB trained with ResNet-152 features while providing an explicitly visible reasoning procedure.The results are reported on the VQA test-dev set.
6. Conclusion
N2NMNs handle complex visual question answering by learning question-specific modular layouts and optimizing both modules and layout policy end-to-end.
- N2NMNs decompose textual questions into subtasks connected by neural modules and predict a suitable layout with a sequence-to-sequence RNN.Training can begin with behavioral cloning from expert layouts and continue with reinforcement learning.
- End-to-end optimization of the neural modules and layout policy yields significant further improvement over behavioral cloning from expert layouts.
- The model demonstrates capability on complicated reasoning problems in visual question answering.