Source-linked AI summary
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, Joshua B. Tenenbaum
TL;DR
VQA systems need to handle challenging compositional reasoning while using limited supervision and remaining interpretable. NS-VQA separates visual and language parsing from symbolic program execution, achieving near-perfect CLEVR performance with compact, transparent reasoning. The paper also identifies generalization to truly novel situations as an ongoing challenge.
Problem
Pure neural VQA models struggle on challenging reasoning tasks, while prior program-based systems require many annotated examples and have difficult-to-explain neural executors.
Method
NS-VQA uses neural parsers to recover a structural scene representation and a question program, then applies a symbolic executor to produce the answer.
Results
99.8% accuracy is achieved on CLEVR using 270 program annotations and 4K images, with near-perfect performance across five question types.
Takeaways & Limitations
Symbolic execution supports robust long-trace reasoning, lower data and memory requirements, and transparent step-by-step diagnosis.
Takeaways & Limitations
Generalizing structured scene and sentence representations to truly novel situations remains a challenge for NS-VQA and related approaches.
Abstract
from arXiv · showhide
We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.
1 Introduction
The paper proposes NS-VQA, which separates visual and language understanding from symbolic reasoning. This design targets difficult VQA reasoning while improving data efficiency, memory efficiency, and interpretability.
- Motivation: Pure neural VQA models often perform poorly on challenging compositional reasoning tasks such as CLEVR.CLEVR uses intricate questions generated by programs to test reasoning ability.
- Approach: NS-VQA uses neural networks to recover structural scene representations and question programs, then executes those programs symbolically to answer questions.The symbolic executor operates on the scene representation rather than directly on visual features.
- Results: 99.8% accuracy is achieved on CLEVR using 270 program annotations and 4K images.The result is described as near-perfect and reflects performance after training with limited annotations and images.
- Advantages: Symbolic representations provide robustness to long, complex program traces and reduce the need for training data.The paper attributes these advantages to executing reasoning programs in a symbolic space.
- Advantages: The compact structural image representation reduces memory cost by 99% compared with other state-of-the-art algorithms.Both the reasoning module and visual scene representation are described as lightweight.
- Advantages: Symbolic scene representations and program traces make the reasoning process transparent, enabling step-by-step analysis and diagnosis.The model must recover underlying programs from questions, exposing the execution process.
2 Related Work
Related work spans interpretable scene representations, program induction from language, and neural VQA systems. These lines of research provide structural and program-based tools for visual reasoning while also exposing concerns about generalization and dataset bias.
- Structural scene representation: Prior work learned interpretable, disentangled representations for visual factors such as facial pose, lighting, and chair pose or content.These approaches use neural networks to infer structured factors from images.
- Program induction from language: Program-induction research has used program search and neural networks to recover programs from domain-specific languages and sentences.Semantic parsing methods map language to logical forms through knowledge bases or programs.
- NS-VQA pipeline: Figure 2 depicts a three-stage pipeline: image segmentation and scene recovery, question-to-program conversion, and program execution.The stages correspond to the scene parser, question parser, and executor.
- Visual question answering: VQA combines semantic and visual understanding, with leading approaches using neural attention to select image regions relevant to questions.Other work explicitly incorporates structural knowledge into reasoning.
- Visual reasoning: CLEVR was designed to control dataset bias and benchmark visual reasoning, while subsequent models combined program generation with attentive execution.The benchmark uses carefully controlled questions to test reasoning capabilities.
3 Approach
NS-VQA separates scene parsing, question-to-program parsing, and symbolic execution into distinct components. Neural modules recover structured inputs, while deterministic functional modules execute programs to produce answers.
- System overview: NS-VQA uses a scene parser, question parser, and program executor to transform an image-question pair into an answer.The scene parser produces a structural scene representation, the question parser generates a hierarchical program, and the executor runs it symbolically.
- Scene parsing: The scene parser de-renders images by proposing object segments and classifying each object and its attributes.The resulting representation is structural, disentangled, compact, and rich.
- Question parsing: The question parser is an attention-based seq2seq model whose hierarchical functional program supplies compositional reasoning and generalization.A bidirectional LSTM encodes variable-length questions, while the decoder uses attention over encoded states to generate program tokens.
- Program execution: The executor implements deterministic Python functional modules that correspond one-to-one with program tokens and can be arranged in arbitrary length and order.Modules pass typed outputs sequentially, and the final module produces the answer; type mismatches trigger an error flag and random final-module sampling.
- Training paradigm: The scene-parser networks are trained on 4,000 generated CLEVR images, while reasoning uses supervised pretraining followed by REINFORCE fine-tuning.The question parser first learns from a small set of annotated question-program pairs, then uses answer correctness as the reinforcement reward.
4 Evaluations
Evaluations show that NS-VQA combines strong CLEVR performance with data-efficient program recovery, compact representations, and generalization across attribute compositions, question styles, and visual contexts.
- CLEVR: NS-VQA achieves near-perfect CLEVR accuracy, outperforming other methods across all five question types after pretraining on 270 annotated programs.The reported results are means of three runs, with standard deviation below 0.1% for 270 pretraining programs and beyond.
- Data efficiency: 99.8% accuracy is retained with 9K question-answer pairs when annotated programs are used for both pretraining and REINFORCE.The model outperforms IEP under multiple supervision conditions, including weaker REINFORCE supervision.
- Interpretability and program recovery: 88% program accuracy with 500 annotations shows that NS-VQA recovers latent programs more reliably than IEP under limited supervision.With 9K programs, the model performs almost perfectly on both question answering and program recovery, whereas IEP recovers only half of the programs.
- Generalization: NS-VQA generalizes across unseen attribute combinations and human-generated question styles, although its vanilla model is affected by scene-parser attribute bias.Fine-tuning or grayscale shape recognition improves performance across CLEVR-CoGenT splits, while NS-VQA outperforms IEP on CLEVR-Humans under limited program annotation.
- New scene contexts: On Minecraft, NS-VQA recovers correct programs and answers in a richer visual context, while most errors arise from perceiving heavily occluded objects.Reasoning generally requires weaker initial program signals than on CLEVR, but the question parser retains its parsing capability.
5 Discussion
NS-VQA separates learned visual and language representations from symbolic reasoning, improving performance, data and memory efficiency, and interpretability while leaving generalization to novel structures challenging.
- NS-VQA uses deep learning for inverse graphics and inverse language modeling, then applies a symbolic program executor to reason and answer questions.
- Symbolic representation as prior knowledge increases performance, reduces annotated-data and memory requirements, and makes reasoning fully interpretable.
- Generalizing structured scene and sentence representations to truly novel situations remains a challenge for NS-VQA and many related approaches.
A Scene Parser Details
The scene parser is trained on rendered CLEVR-style images with object masks and attributes, then uses Mask-RCNN detection and segmentation proposals to learn object features.
- 4,000 CLEVR-style Blender-rendered images provide object masks and ground-truth color, material, shape, size, and 3D-coordinate attributes for scene-parser training.
- For CLEVR-CoGenT, another image set is generated with the required attribute-composition restrictions using the same rendering software.
- Mask-RCNN is trained on the rendered images and masks, with a 48-class bounding-box classifier representing combinations of shapes, materials, and colors.
- Detector-generated segmentation proposals are paired with labeled objects to create segment-label training pairs for subsequent feature learning.
B Program Executor Details
The program executor consists of functional modules that sequentially apply designated logic operations to abstract scene representations, with tables organizing module types.
- The executor runs functional Python modules one by one, passing each module’s output iteratively to the next in a program sequence.
- Executor inputs and outputs include scenes, objects, and attribute entries within an abstract scene representation.
- The module reference covers set operations, as identified by Table 3.
- The module reference separately includes Boolean, query, relation, and filter operations in Tables 4–7.
C Running Examples
The running examples illustrate NS-VQA execution traces, including intermediate scene, object, and attribute outputs, module-output joining, and a spurious-program failure case.
- Intermediate execution outputs can be a scene, a single object, or an attribute entry such as “blue” or “rubber.”
- Dashed arrows indicate that outputs from preceding program modules are joined and sent to the next module.
- The examples present a running NS-VQA execution trace across successive illustrated stages.
- A spurious program can produce the correct answer while adding extra degenerate structures relative to the ground-truth logic.
D Scene Parsing on Real Images
Scene parsing transfers to handcrafted real-world CLEVR objects without fine-tuning, correctly detecting and extracting attributes from most objects despite their mismatch with synthetic scenes.
- Real-image scene parsing: Without fine-tuning, the model applies scene parsing to handcrafted real-world CLEVR objects that differ from the synthetic scenes.The real objects were made with paper boxes and rolls and were not well aligned with the synthetic scenes.
- Real-image scene parsing: The model detects and extracts attributes from most real objects correctly.
- Real-image scene parsing: In some cases, the model mistakenly treats shadows as objects.