Source-linked AI summary
Learning by Abstraction: The Neural State Machine
Drew A. Hudson, Christopher D. Manning
TL;DR
Visual reasoning requires models that can generalize compositionally while remaining interpretable and modular. The Neural State Machine constructs a probabilistic semantic graph, translates visual and linguistic information into shared concepts, and performs sequential graph traversal; it achieves state-of-the-art results on GQA and VQA-CP with evidence of broad generalization.
Problem
Neural networks can struggle with systematic generalization and causal relations, while their statistical scale limits interpretability, modularity, and soundness in visual reasoning.
Method
The Neural State Machine constructs a probabilistic scene graph and simulates an automaton by recurrently traversing graph states under question-derived instructions.
Results
The model achieves state-of-the-art results on GQA and VQA-CP and generalizes across novel concept compositions, answer distributions, and unseen linguistic structures.
Takeaways & Limitations
Shared semantic concepts and sequential graph computation integrate abstraction and compositionality while supporting modularity, versatility, and generalization.
Abstract
from arXiv · showhide
We introduce the Neural State Machine, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning. Given an image, we first predict a probabilistic graph that represents its underlying semantics and serves as a structured world model. Then, we perform sequential reasoning over the graph, iteratively traversing its nodes to answer a given question or draw a new inference. In contrast to most neural architectures that are designed to closely interact with the raw sensory data, our model operates instead in an abstract latent space, by transforming both the visual and linguistic modalities into semantic concept-based representations, thereby achieving enhanced transparency and modularity. We evaluate our model on VQA-CP and GQA, two recent VQA datasets that involve compositionality, multi-step inference and diverse reasoning skills, achieving state-of-the-art results in both cases. We provide further experiments that illustrate the model's strong generalization capacity across multiple dimensions, including novel compositions of concepts, changes in the answer distribution, and unseen linguistic structures, demonstrating the qualities and efficacy of our approach.
1 Introduction
The paper motivates the Neural State Machine as a bridge between neural and symbolic AI for visual reasoning. It represents image semantics and question-guided reasoning in an abstract, compositional structure intended to improve modularity, transparency, and generalization.
- Humans abstract sensory experiences into compositional world models and use conceptual rules to derive novel conclusions.
- Neural networks often struggle with systematic generalization and causal understanding, while their scale and statistical nature hinder interpretability, modularity, and soundness.
- The Neural State Machine first generates a probabilistic scene graph from an image, then performs question-guided sequential reasoning over that graph.
- Semantic embedded concepts provide a shared vocabulary for visual scene graphs and linguistic reasoning instructions, separating structure from content in an abstract space.
- The model achieves state-of-the-art results on GQA and VQA-CP and shows generalization across novel concept compositions and unseen linguistic structures.
- The approach combines structured semantic representations with iterative computation over a differentiable state machine to support abstraction and compositionality.
2 Related work
The Neural State Machine builds on research in compositionality, concept acquisition, and neural computation while extending visual reasoning toward structured, iterative inference. Its closest predecessor is MAC, but NSM reasons over semantic graphs rather than spatial feature maps.
- The model relates to work on compositionality, concept acquisition, neural computation, and structural priors for bridging symbolic and neural paradigms.
- Prior VQA systems commonly use dense CNN or object-detector features, sometimes augmented with contextual object relationships.
- Compared with MAC, NSM traverses graph nodes and edges and uses shared semantic concepts for visual and linguistic knowledge instead of raw observational features.
3 The Neural State Machine
The Neural State Machine converts images into probabilistic semantic graphs and questions into concept-based instructions, then performs differentiable sequential reasoning by traversing the graph. Its shared concept vocabulary and state-machine formulation provide structured, abstract representations for visual reasoning.
- Overview: The model has modeling and inference stages: it constructs a probabilistic scene graph and then simulates reasoning over it.The graph represents image semantics, while question-derived instructions guide computation over the graph.
- Formal machine: The neural state machine represents states with an initial probability distribution and recurrently updates that distribution for N computation steps.Unlike deterministic finite automata, it maintains probabilistic state attention during computation.
- Concept vocabulary: Its alphabet is a learned vocabulary of embedded concepts covering object identities, attributes, and relations.The vocabulary provides a shared semantic basis for visual representations and reasoning instructions.
- States and edge transitions: A probabilistic scene graph uses object nodes as states and relation edges as valid transitions, with semantic distributions attached to both.Object nodes include visual and property information, while relation edges encode probable semantic types.
- States and edge transitions: Structured node and edge representations factor raw visual information into distributions over disentangled semantic concepts.This soft-binding of concepts across semantic properties is intended to encourage compositional representations.
- Reasoning instructions: Question words are tagged against the concept vocabulary, replacing relevant content words with concepts while retaining unmatched function words near their original embeddings.The resulting sequence of concept-based instructions guides graph reasoning.
- Model simulation: During simulation, each instruction determines relevant node and edge properties, shifts attention toward neighboring states, and yields sequential reasoning over the scene graph.The process grounds successive instructions in graph nodes and edges until an answer is reached.
4 Experiments
The experiments evaluate the Neural State Machine on GQA and VQA-CP, including single-model performance and generalization to changed answer distributions, unseen concepts, and unseen linguistic structures. NSM achieves state-of-the-art or significantly stronger results across these settings.
- GQA evaluation: NSM achieves state-of-the-art single-model performance across GQA’s accuracy and consistency metrics.The comparison includes official baselines and top submissions.
- GQA evaluation: NSM’s GQA ensemble ranks third among 52 challenge submissions.The ensemble combines predictions from 10 model instances.
- GQA evaluation: NSM learns GQA directly from question-answer pairs without using the dataset’s functional programs as training supervision.This tests whether the model can handle diverse reasoning forms, especially multi-step inference.
- Generalization experiments: The GQA generalization studies test changed answer distributions, concepts learned only in isolation, and unseen grammatical structures.The content and structure splits are constructed using question grounding annotations.
- VQA-CP evaluation: NSM surpasses alternative models by a large margin on VQA-CP under the standard VQA accuracy metric.VQA-CP changes the answer distribution between training and test sets to reduce reliance on dataset biases.
- Generalization experiments: NSM performs significantly better than GQA baselines on both unseen-concept and unseen-structure splits.The results support generalization to concepts without question training and to novel linguistic structures.
5 Conclusion
The Neural State Machine integrates structured graph representations with sequential neural reasoning for visual reasoning and compositional question answering. It demonstrates versatility, robustness, and high generalization skills across these tasks.
- The Neural State Machine is a graph-based network that simulates the operation of an automaton.
- A structured graph serves as the world model, while sequential reasoning over its nodes enhances compositional computation.
- The model aims to bring symbolic and connectionist approaches closer together for higher-level abstraction, knowledge representation, compositionality, and reasoning.
7 Supplementary material
The supplementary material details the model’s structured representations, scene-graph construction, related-work distinctions, evaluation settings, and ablations. These experiments show that graph structure, sequential traversal, semantic concepts, and sufficient computation steps each contribute to performance.
- Results: NSM surpasses baselines across all GQA structural and semantic question types, with largest gains on relational and comparative questions.
- Related work: NSM uses a fully-neural graph-based design that scales to real-world settings, unlike related approaches relying on symbolic execution or task-specific modules.
- Related work: NSM differs from MAC by traversing scene-graph nodes and edges while representing visual and linguistic information with shared semantic concepts.
- Ablation studies: 58.48% accuracy with dense graph-element features versus 62.95% with concept-based representations demonstrates the significance of semantic representations.
- Ablation studies: Accuracy steadily increases with more reasoning steps until saturation at N = 8 steps, supporting sequential computation for compositional questions.
- Model representation: The concept vocabulary contains 785 objects, 170 relations, and 303 attributes divided into 77 types, derived from Visual Genome annotations.
- Scene-graph construction: Scene-graph generation detects objects and relations, using proximity-based edges that cover over 94% of ground-truth edges while sparsifying computation.