Source-linked AI summary
Compositional Attention Networks for Machine Reasoning
Drew A. Hudson, Christopher D. Manning
TL;DR
The paper addresses how neural networks can perform structured, iterative reasoning without sacrificing end-to-end learning and versatility. It introduces MAC, which decomposes problems into attention-based steps using recurrent cells with separate control and memory, and reports state-of-the-art CLEVR results with strong data efficiency. The paper also identifies error patterns on CLEVR and CLEVR-Humans, including counting, occlusion, and reasoning skills absent from training.
Problem
Neural networks often struggle with compositional tasks and can rely on spurious statistical patterns, while symbolic approaches require externally supplied structures, brittle parsers, or complex training procedures.
Method
MAC is an end-to-end differentiable architecture that chains recurrent cells separating control from memory to infer and execute multi-step reasoning operations.
Results
98.9% accuracy is achieved on CLEVR, with state-of-the-art performance across reasoning tasks and settings and strong generalization from substantially less data.
Takeaways & Limitations
The results support imposing structural priors that encourage compositional reasoning while retaining end-to-end differentiability and inferred reasoning steps.
Takeaways & Limitations
CLEVR-Humans errors often involve physical properties, relative distances and amounts, commonalities and uniqueness, or negative questions that the model was not trained for.
Abstract
from arXiv · showhide
We present the MAC network, a novel fully differentiable neural network architecture, designed to facilitate explicit and expressive reasoning. MAC moves away from monolithic black-box neural architectures towards a design that encourages both transparency and versatility. The model approaches problems by decomposing them into a series of attention-based reasoning steps, each performed by a novel recurrent Memory, Attention, and Composition (MAC) cell that maintains a separation between control and memory. By stringing the cells together and imposing structural constraints that regulate their interaction, MAC effectively learns to perform iterative reasoning processes that are directly inferred from the data in an end-to-end approach. We demonstrate the model's strength, robustness and interpretability on the challenging CLEVR dataset for visual reasoning, achieving a new state-of-the-art 98.9% accuracy, halving the error rate of the previous best model. More importantly, we show that the model is computationally-efficient and data-efficient, in particular requiring 5x less data than existing models to achieve strong results.
1 INTRODUCTION
The paper proposes MAC, a fully differentiable architecture for explicit, structured reasoning that balances end-to-end versatility with more transparent computation. It evaluates MAC on CLEVR, a compositional visual-question-answering benchmark designed to require reasoning rather than dataset shortcuts.
- Task: CLEVR uses unbiased, highly compositional questions requiring transitive and logical relations, counting, and comparisons without reasoning shortcuts.The task tests visual question answering over images and natural-language questions.
- Motivation: Deep learning models often struggle with compositional, structured tasks because their statistical nature can limit interpretability and explicit inference.Symbolic alternatives may require external programs, brittle parsers, expert demonstrations, or complex reinforcement-learning training.
- Approach: MAC sequences recurrent Memory, Attention and Composition cells to perform structured reasoning through inferred multi-step operations.The architecture maintains separate control and memory, while its control, read, and write units coordinate question attention, knowledge retrieval, and memory updates.
- Results: 98.9% accuracy is achieved on CLEVR, halving the error rate of the previous best model.The abstract reports this as a new state-of-the-art result on the visual-reasoning dataset.
- Results: MAC achieves state-of-the-art accuracy across CLEVR reasoning tasks and settings, including more difficult human-authored questions.The model performs particularly well on counting and aggregation questions and generalizes effectively from an order of magnitude less data than other approaches.
2 THE MAC NETWORK
The MAC network performs explicit multi-step reasoning by sequencing recurrent MAC cells that separate question-driven control from image-based memory. Its control, read, write, and output units iteratively select operations, retrieve information, update intermediate results, and predict answers.
- Architecture: A MAC network sequences p recurrent MAC cells, each responsible for one explicit reasoning step over a knowledge base and task description.The architecture comprises input, recurrent MAC-cell, and output units.
- MAC Cell: Each MAC cell maintains dual hidden states: control ci for the current reasoning operation and memory mi for intermediate results.The states are initialized with learned parameters and have dimension d.
- Operational Units: The write unit integrates retrieved information with the preceding memory under the current control, producing a new intermediate result for the next reasoning step.This recurrent update forms the cell’s evolving memory state.
- Operational Units: The control unit attends to question words to identify the next operation, while the read unit retrieves image information relevant to that operation and prior memory.The read unit represents retrieved information ri as a weighted average over knowledge-base regions.
- Structural Constraint: Control and memory interact only through interpretable attention distributions and sigmoid gates, preserving separate textual and visual representational spaces.The paper links this separation to improved generalizability and transparency.
- Illustrative Reasoning: A three-step example traverses from a tiny blue block to a cyan sphere and then a purple cylinder, enabling the model to answer the question correctly.Each step combines the current control with previously stored visual results.
- Output: The output unit predicts the answer from the question representation and final memory using a two-layer fully connected softmax classifier for CLEVR.The classifier produces a distribution over the fixed set of candidate answers.
3 RELATED WORK
Prior CLEVR models largely fall into module networks with strong program supervision or relatively unstructured end-to-end differentiable networks augmented for reasoning.
- Model Categories: CLEVR approaches are broadly divided into module networks and relatively unstructured end-to-end differentiable networks.Module networks use structured functional programs accompanying each data instance, while end-to-end models combine CNNs with reasoning components.
- Module Networks: Module-network approaches generally rely on strong supervision from structured functional programs.
- End-to-End Networks: End-to-end approaches typically augment a standard CNN stack with components intended to support reasoning tasks.
4 EXPERIMENTS
Experiments show that MAC performs strongly across CLEVR reasoning settings, learns efficiently from less data, and benefits from its structured attention and separated control-memory design. Qualitative analyses indicate focused, interpretable reasoning over question terms and image regions.
- 98.94% overall accuracy on the primary 700k CLEVR dataset more than halved the error rate of the best prior model, FiLM.
- MAC nearly halved the error rate on counting and numerical-comparison questions, where existing models consistently struggled.Attention supports reasoning operations requiring global aggregation across image regions.
- 81.5% accuracy on CLEVR-Humans surpassed the next-best model by 5.6%, before and after fine-tuning.The dataset contains natural-language questions with diverse vocabulary and linguistic variations.
- Ablations: Question attention improved learning and generalization: removing it reduced overall accuracy by 18.5%, while unconstrained control states slowed convergence 6x.The ablations support decomposing questions into sequences of simpler operations.
- Interpretability: Attention maps focused on relevant objects and structural keywords, providing evidence of transitive reasoning and multi-step counting or summation.The model’s visual and linguistic attentions often tracked concrete entities and words such as “or,” “and,” and “how many.”
5 CONCLUSION
The conclusion presents MAC as an end-to-end differentiable architecture for explicit multi-step machine reasoning. It reports strong CLEVR performance, low-data generalization, adaptation to diverse language, and interpretable attention-based rationales.
- MAC chains recurrent cells that separate memory and control to perform inferred reasoning steps successively.
- The model achieves state-of-the-art CLEVR visual-reasoning results and generalizes well from a 10% data subset.
- Quantitative and qualitative studies show adaptation to novel situations and diverse language alongside interpretable attention-based rationales.
A IMPLEMENTATION AND TRAINING DETAILS
Training uses Adam optimization with regularization and validation-based early stopping. The reported setup trains the model for 10–20 epochs on a single Maxwell Titan X GPU.
- Adam training uses a 10^-4 learning rate, batch size 64, gradient clipping, and validation-accuracy early stopping.
- Training lasts 10–20 epochs, equivalent to roughly 15–30 hours on a single Maxwell Titan X GPU.
- Word vectors have dimension 300 and are initialized randomly using a standard uniform distribution.
B ERROR ANALYSIS
The error analysis identifies occlusion-related recognition and counting mistakes on CLEVR, while CLEVR-Humans exposes broader failures on unseen reasoning skills and word meanings.
- Occlusions make it harder to recognize objects’ materials or shapes, contributing to many CLEVR errors.
- Off-by-one counting mistakes frequently occur when questions require summing two groups of objects.
- Heavily occluded objects can lead to slight underestimation, suggesting a form of continuous rather than discrete counting.
- About half of CLEVR-Humans errors involve reasoning skills absent from CLEVR training, including physical properties, distances, amounts, commonalities, and negative questions.
- Some CLEVR-Humans errors reflect plausible but incorrect interpretations of unseen words, colors, or object categories.
C ABLATION STUDIES
Ablations show that MAC benefits from longer recurrent reasoning, shared adaptive cells, question-conditioned control, and enhanced memory writing, with data availability affecting gate-bias choices.
- Network Length: Network performance improves significantly with reasoning length up to p = 8, indicating effective use of recurrent cells for compositional reasoning.
- Weight Sharing: Weight sharing helps on both the primary CLEVR task and limited-data settings, suggesting one MAC cell can adapt across reasoning steps.
- Control Unit: 51.1% accuracy results when the control unit reasons over the image without question access, while question-word attention improves step decomposition.
- Control Unit: Contextual words from a biLSTM yield better performance and faster learning than directly attending to learned word vectors.
- Write Unit: 99.48% accuracy results when self-attention and memory gating are combined, compared with 98.94% for the basic MAC model.
- Write Unit: Ignoring prior memory content reduces accuracy by only 0.4%, making direct assignment variants only slightly worse than the basic write unit.
- Gate Bias Initialization: A gate bias of 1 is optimal with the full dataset, whereas 0 is ideal with 10% of the data.
D RELATED WORK
Related CLEVR methods divide into program-supervised module networks and end-to-end differentiable CNN-based networks, with the comparison organized around these approaches.
- CLEVR models are partitioned into module networks using structured functional programs and end-to-end differentiable networks built around CNNs.
- The section frames these groups as alternative strategies for adding reasoning capabilities to visual question answering systems.
D.1 MODULE NETWORKS
MAC differs from module and other end-to-end approaches through a universal adaptive cell, sequential reasoning, and selective question-guided attention.
- Module Networks: Module networks translate questions into externally supervised tree-structured plans and execute them with specialized predefined modules.
- Module Networks: MAC replaces specialized modules with one universal cell whose shared parameters and architecture adapt across reasoning steps.
- Module Networks: MAC also replaces dynamic recursive trees with a sequential topology augmented by soft attention.
- Relation Networks: Relation Networks inspect all pixel pairs to model binary relations but have quadratic computational complexity, unlike MAC’s linear approach.
- FiLM: FiLM conditions CNN layers through question-derived bias and variance terms, whereas MAC uses attention to focus on question-relevant image regions.
- FiLM: The paper states that selective question–image interaction facilitates learning and increases generalizability.
D.3 MEMORY AND ATTENTION
MAC separates question-driven control from recurrent memory while iteratively attending to relevant information. Attention maps illustrate tracking, logical operations, counting, and summation across reasoning steps.
- Memory and attention: Unlike DMN, MAC decomposes the question into a multi-step action plan that gives each reasoning cell a specific objective.DMN treats the question as one atomic unit, whereas MAC maintains distant interaction between question and knowledge base.
- Memory and attention: MAC uses a recurrent memory structure in which each cell has its own memory state, rather than repeatedly reading and writing shared memory slots.
- Attention maps: Attention maps from MAC networks of lengths 4 and 6 show iterative tracking of transitive relations and logical operations, often proceeding backward from the question.
- Attention maps: A length-6 MAC network attends first to structural words such as “many” and “or,” then performs counting, logical operations, and summation over object groups.The examples include correctly summing two object groups.