Source-linked AI summary
Inferring and Executing Programs for Visual Reasoning
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, Ross Girshick
TL;DR
Visual reasoning models often map inputs directly to answers without explicitly modeling compositional reasoning, allowing dataset biases to substitute for reasoning. This paper learns a program generator and neural execution engine, trained with supervision, backpropagation, and REINFORCE. On CLEVR and human-posed questions, the model outperforms competing VQA systems and generalizes to novel compositions.
Problem
Direct input-output visual reasoning models do not explicitly formulate compositional plans and tend to learn dataset biases instead of reasoning.
Method
The model generates an explicit program from each question and executes it on the image using a learned neural module network, with training supported by backpropagation and REINFORCE.
Results
The model outperforms state-of-the-art non-compositional VQA models by ∼20 percentage points on CLEVR using 9000 ground-truth programs, and improves ∼9 points over competing models on human-posed questions.
Takeaways & Limitations
Explicit program representations make it easier to compose programs for novel image questions, while the learned generator and modules support extension to new problems and domains.
Takeaways & Limitations
The model underperforms a CNN+LSTM+SA baseline on questions about object shape or color when attribute combinations are novel.
Abstract
from arXiv · showhide
Existing methods for visual reasoning attempt to directly map inputs to outputs using black-box architectures without explicitly modeling the underlying reasoning processes. As a result, these black-box models often learn to exploit biases in the data rather than learning to perform visual reasoning. Inspired by module networks, this paper proposes a model for visual reasoning that consists of a program generator that constructs an explicit representation of the reasoning process to be performed, and an execution engine that executes the resulting program to produce an answer. Both the program generator and the execution engine are implemented by neural networks, and are trained using a combination of backpropagation and REINFORCE. Using the CLEVR benchmark for visual reasoning, we show that our model significantly outperforms strong baselines and generalizes better in a variety of settings.
1. Introduction
Visual reasoning requires compositional plans for novel combinations, whereas direct input-output models often exploit dataset biases. The paper introduces a learned program generator and execution engine, achieving strong CLEVR and human-question results.
- Compositional reasoning lets models understand novel object interactions by combining basic concepts such as person, bike, and touching.
- Direct input-output models fail on tasks spanning large spaces of objects, attributes, actions, and interactions, and tend to learn dataset biases instead.
- The proposed model generates question-specific programs from a function dictionary and executes them through neural modules on the image.
- ∼20 percentage points improvement over state-of-the-art non-compositional VQA models is achieved on CLEVR using only 9000 ground-truth programs, or 2% of those available.
- ∼9 point improvement over the best competing models is obtained after finetuning on human-posed free-form questions without additional program supervision.
2. Related Work
The paper distinguishes its learned visual reasoning system from prior reasoning-augmented, module-network, semantic-parsing, and program-induction approaches.
- VQA combines question and image representations to classify answers, but benchmark performance can arise from dataset biases rather than visual reasoning.
- Reasoning-augmented models add components such as explicit memory to neural networks, whereas this work focuses on compositional visual reasoning.
- Unlike module networks using hand-designed syntactic parsers, this model learns a task-adapted program generator.
- Unlike semantic parsers, the paper learns both the program generator and execution engine rather than assuming fixed program semantics and execution.
- Unlike prior neural program interpreters for simple algorithms, this approach handles image-and-question inputs and uses a learned execution engine with minimal prior knowledge.
3. Method
The method predicts an explicit program from each question, assembles a matching neural module network over image features, and trains the components with supervision, backpropagation, and REINFORCE.
- The model maps an image-question pair to a program representing reasoning steps, then executes that program to produce an answer distribution.
- 3.2. Program generator: The program generator π(q) predicts function sequences from questions, using prefix traversal to serialize syntax trees for sequence-to-sequence modeling.
- 3.2. Program generator: At test time, argmax function choices form a sequence converted into a syntax tree, with padding or discarded functions handling invalid lengths.
- 3.3. Execution engine: The execution engine φ(x,z) assembles a question-specific neural module network whose structure mirrors the predicted program.
- 3.3. Execution engine: Using one generic architecture for all modules ensures every valid program becomes a neural network producing features for answer classification.
- 3.4. Training: Without ground-truth programs, sampling replaces argmax decoding and REINFORCE estimates gradients using the execution engine’s negative zero-one-loss reward.
- 3.4. Training: Semi-supervised training initializes the program generator with a small set of ground-truth programs, trains the engine on predicted programs, then jointly finetunes both components.
4. Experiments
Experiments evaluate the model under strong and limited program supervision, then test its compositional generalization, learned module behavior, and transfer to new language and attribute combinations.
- Evaluation setup: CLEVR provides ground-truth programs and serves as the primary benchmark for comparing the model with reproduced VQA baselines.The experiments include strong supervision, semi-supervised learning, and several generalization settings.
- Strong and semi-supervised learning: Strong supervision yields near-perfect CLEVR accuracy, outperforming Mechanical Turk workers.The program generator and execution engine are trained separately using ground-truth programs for all questions.
- Strong and semi-supervised learning: 18K ground-truth programs are sufficient for performance almost on par with full supervision, while predicted-program training initially costs about 3 accuracy points.Joint finetuning mitigates some of the loss from replacing ground-truth programs with predicted programs.
- What do the modules learn?: The model attends to correct objects in complex referring expressions, and changing one module changes both attention and predicted answers.Modules learn functions such as localization and set operations without explicit supervision of their outputs.
- Generalizing to new attribute combinations: On CLEVR-CoGenT, the model learns new attribute combinations from about 10K questions and about 1K images, but remains weak on shape- and color-query questions in Condition B.The reported limitation arises when attribute combinations such as red cubes are absent during training.
- Generalizing to new question types: Finetuning on long questions substantially improves performance after short-question training, while finetuning on human questions reuses learned skills and outperforms all baselines.The model can map novel words to known modules but often fails when questions require skills outside its module inventory.
5. Discussion and Future Work
The model generalizes to novel scenes, questions, and free-form human queries, but its fixed module vocabulary limits the questions it can reasonably answer.
- The model generalizes to novel scenes and questions and infers programs for free-form human questions using learned modules.
- Questions requiring capabilities absent from the fixed module set remain difficult, such as identifying unique shapes.The paper gives the question “What color is the object with a unique shape?” as an example.
- Adding modules is straightforward with the generic design, but automatically identifying and learning new modules without program supervision remains an open problem.The paper suggests Turing-complete modules with control-flow operators as one possible direction.
6. Conclusion
The paper argues that explicit program representations, a learnable program generator, and universal modules support composing programs for novel image questions and make the model extensible.
- Explicit program representations make it easier to compose programs for answering novel questions about images.
- A generic program representation, learnable program generator, and universal module design make the model more flexible and extensible than neural module networks.
A. Implementation Details
The paper states that code for reproducing the experiments will be released and introduces implementation details afterward.
- The authors will release code to reproduce their experiments.
A.1. Program Generator
The program generator is an LSTM sequence-to-sequence model that encodes questions and decodes function sequences, with learned training parameters specified for supervised training.
- The program generator uses separate learned recurrent encoder and decoder networks in an LSTM sequence-to-sequence architecture.The encoder summarizes the question into a fixed-length vector, and the decoder produces the program as a function sequence.
- The encoder maps question words to 300-dimensional embeddings and processes them with a two-layer LSTM using 256 hidden units per layer.The final hidden state of the second layer feeds the decoder.
- At each decoding step, the model combines the previous function embedding with the encoder output before processing them through a two-layer LSTM decoder.The decoder uses 300-dimensional function embeddings and 256 hidden units per layer.
- Supervised program-generator training uses Adam with learning rate 5 × 10^-4, batch size 64, and early stopping within 32,000 iterations.
A.2. Execution Engine
The execution engine turns a predicted syntax-tree program into a custom neural module network that processes image features and predicts an answer distribution.
- The execution engine uses a Neural Module Network to compile an architecture from the program generator’s predicted program.
- The predicted program is a syntax tree whose Scene leaves receive visual features from the convolutional network.
- Each program function maps to a module receiving one or two inputs, producing a custom architecture for that program.
- Modules use a generic small residual-block design rather than architectures individually customized to function types.
- Batch Normalization was removed because minibatch programs are executed individually, making module batch sizes one and harming convergence.
- The final module output enters a classifier that predicts a distribution over answers.
A.3. Joint Training
Joint training optimizes the program generator and execution engine together, while REINFORCE variance is reduced with a moving-average reward baseline.
- Joint training uses Adam with learning rate 5 × 10^-5, batch size 64, a maximum of 100,000 iterations, and validation-based early stopping.
- A moving-average baseline reduces the variance of REINFORCE gradient estimates.
- The baseline is an exponentially decaying average of past rewards with decay factor 0.99.
- Unary execution modules receive one module output, while binary modules receive two module outputs.
- Binary module functions include intersection, union, equality comparisons, and ordering comparisons.
- The classifier consumes the final module output and predicts a distribution over answers.
A.4. Baselines
The baselines comprise recurrent, convolutional-recurrent, and stacked-attention models; adding an MLP substantially improves the strongest attention baseline on CLEVR.
- The LSTM baseline encodes the question with learned 300-dimensional embeddings and a two-layer LSTM before an MLP answer classifier.
- CNN+LSTM concatenates question and image representations before passing them to a two-layer MLP.
- CNN+LSTM+SA applies two consecutive stacked-attention layers to the question and image representations before predicting answer scores.
- CNN+LSTM+SA+MLP adds a two-layer MLP after stacked attention for a fairer comparison with models terminating in MLP classifiers.
- 73.2 overall accuracy is achieved by CNN+LSTM+SA+MLP versus 69.8 for CNN+LSTM+SA on CLEVR.
- 69.7 shape-comparison accuracy is achieved by CNN+LSTM+SA+MLP versus 50.9 for CNN+LSTM+SA.
B. Neural Module Network parses
The prior neural module network parser uses dependency parsing and heuristics to construct candidate layouts, but it often fails on long CLEVR questions and misses key question elements.
- Dynamic neural module networks derive layout fragments from dependency parses, heuristically combine them, and learnedly rerank candidate layouts.
- The prior parser performed poorly on the longer questions in CLEVR.
- Table 8 presents random CLEVR training questions and their semicolon-separated parser layout fragments, including fallback outputs.
- Parser failures trigger a default “what thing” fragment, producing module networks that do not respect the question’s structure.
- Even without fallback, layout fragments can omit key question elements, such as the cylinder in a question about a big purple cylinder.