Source-linked AI summary
Explainable and Explicit Visual Reasoning over Scene Graphs
Jiaxin Shi, Hanwang Zhang, Juanzi Li
TL;DR
Complex visual reasoning systems can exploit dataset and vision-to-reasoning shortcuts, motivating a more explicit separation between perception and reasoning. The paper proposes XNMs, reusable attention-based modules that reason over scene graphs of varying quality. XNMs reach 100% accuracy with ground-truth graphs on CLEVR and CLEVR-CoGenT, while their practical performance depends on scene-graph quality.
Problem
End-to-end visual reasoning models can learn dataset shortcuts, while existing neural modules remain coupled to visual perception and can fail under novel object attributes.
Method
XNMs perform explicit, explainable reasoning over scene graphs using four reusable attention-based meta-types while supporting different graph representations.
Results
100% accuracy is achieved with ground-truth scene graphs and programs on both CLEVR and CLEVR-CoGenT, while detected-graph performance on CoGenT Condition B drops to 72.1%.
Takeaways & Limitations
XNMs make reasoning flow explicitly traceable through graph attentions and indicate that visual reasoning benefits substantially from high-quality scene graphs.
Takeaways & Limitations
The approach relies on scene graphs whose detection quality varies, and its implementation requires differentiable functions with outputs in [0, 1].
Abstract
from arXiv · showhide
We aim to dismantle the prevalent black-box neural architectures used in complex visual reasoning tasks, into the proposed eXplainable and eXplicit Neural Modules (XNMs), which advance beyond existing neural module networks towards using scene graphs --- objects as nodes and the pairwise relationships as edges --- for explainable and explicit reasoning with structured knowledge. XNMs allow us to pay more attention to teach machines how to "think", regardless of what they "look". As we will show in the paper, by using scene graphs as an inductive bias, 1) we can design XNMs in a concise and flexible fashion, i.e., XNMs merely consist of 4 meta-types, which significantly reduce the number of parameters by 10 to 100 times, and 2) we can explicitly trace the reasoning-flow in terms of graph attentions. XNMs are so generic that they support a wide range of scene graph implementations with various qualities. For example, when the graphs are detected perfectly, XNMs achieve 100% accuracy on both CLEVR and CLEVR CoGenT, establishing an empirical performance upper-bound for visual reasoning; when the graphs are noisily detected from real-world images, XNMs are still robust to achieve a competitive 67.5% accuracy on VQAv2.0, surpassing the popular bag-of-objects attention models without graph structures.
1. Introduction
XNMs address the shortcut-prone nature of complex visual reasoning by separating visual perception from explicit reasoning over scene graphs. Their four reusable meta-types produce traceable graph attentions and support varied graph qualities.
- Deep end-to-end models perform poorly on high-level vision-language tasks because they can learn dataset shortcut bias instead of reasoning.
- Existing neural module networks make reasoning more transparent through compositional modules, but their visual perception remains attached to reasoning.
- XNMs insulate low-level visual perception from reasoning by operating on scene graphs whose nodes represent entities and edges represent relationships.
- The four XNM meta-types are AttendNode, AttendEdge, Transfer, and Logic, producing graph attentions that are traceable and reusable.AttendNode finds entities, AttendEdge finds relationships, Transfer moves node attention along attentive edges, and Logic performs basic operations on attention maps.
- XNMs support different scene-graph qualities and achieve 100% accuracy with ground-truth graphs and programs on CLEVR and CLEVR-CoGenT.The experiments also evaluate noisily detected graphs on visual question answering benchmarks.
2. Related Work
Visual reasoning methods range from holistic feature-fusion models to neural module networks that execute question-derived programs. Scene-graph reasoning adds object relationships to structural representations, extending beyond object-only approaches toward real-world visual question answering.
- Visual Reasoning: Visual reasoning benchmarks test compositional language understanding and elementary reasoning over visual information, with CLEVR as a representative dataset.
- Visual Reasoning: Holistic approaches fuse image and question features, whereas neural module approaches parse questions into programs and execute modules over image features.
- Neural Module Networks: Neural module networks decompose complex questions into subtasks, assemble corresponding modules dynamically, and expose intermediate outputs.
- Neural Module Networks: NS-VQA improved CLEVR accuracy from 99.1% to 99.8% using object-level structural representations, but its missing object relationships limited real-world applications such as VQAv2.0.
- Scene Graphs: Scene graphs represent images through objects and their relationships, but scene-graph detection remains substantially less satisfactory than object detection.
3. Approach
XNMs reason over scene graphs through four reusable, attention-based meta-types that separate visual perception from explicit reasoning. Their graph-attention execution supports different scene-graph qualities and produces traceable intermediate results.
- Scene-graph reasoning: XNMs execute question-derived module programs over scene graphs containing object nodes and directed relationship edges.The pipeline parses the image and question, then reasons over the resulting graph.
- Scene-graph representations: XNMs support ground-truth labeled graphs and detected label-agnostic graphs using visual or coordinate-derived node and edge features.The GT setting uses object and relation label embeddings, while Det uses detected objects, RoI features, and fused or coordinate-based edge features.
- X Neural Modules: The four meta-types are AttendNode, AttendEdge, Transfer, and Logic, covering entity selection, relation selection, attention propagation, and logical operations.AttendNode and AttendEdge produce attentions over nodes and edges; Transfer moves node weights along attended relations; Logic provides And, Or, and Not operations.
- X Neural Modules: XNMs operate explicitly on node and edge attention maps, making intermediate reasoning results traceable, differentiable, and composable into larger modules.The modules can be assembled into composite functions and trained end-to-end.
- Attention functions: Attention functions differ by graph setting: GT uses softmax over label embeddings, whereas Det uses sigmoid-based node and edge attention weights.For GT graphs, corresponding label weights are summed for nodes and edges; in Det, MLPs map visual features to the embedding dimension.
- Feature output modules: Output modules transform intermediate attentions into answer features, with Exist and Count addressing yes/no and counting questions and Describe encoding attributes.These features are passed to answer prediction or further reasoning modules.
- Program generation and training: For datasets with program annotations, an LSTM sequence-to-sequence model converts word sequences into executable module programs; otherwise, soft module selection uses a differentiable stack.The latter approach follows StackNMN for datasets such as VQAv2.0 without layout annotations.
4. Experiments
Experiments evaluate XNMs across CLEVR, CLEVR-CoGenT, and VQAv2.0, testing reasoning accuracy, robustness to graph and program quality, efficiency, convergence, and explainability. XNMs achieve perfect reasoning with ground-truth representations, remain robust under noisy graphs, and expose interpretable attention-based reasoning.
- CLEVR: 100% accuracy on CLEVR follows from using ground-truth scene graphs and programs, establishing an empirical upper-bound for XNM reasoning.The result separates high-level reasoning from low-level perception.
- CLEVR: 97.9% accuracy with noisy detected scene graphs and ground-truth programs demonstrates robustness to varying graph quality.Replacing ground-truth programs with parsed programs reduces accuracy by 0.1% in both GT and Det settings.
- CLEVR: 0.22M parameters in the GT setting makes XNMs substantially smaller than existing neural module networks.Reusable X modules avoid bundling every module with its input.
- CLEVR: 100% accuracy with only 10% of the training data shows strong data efficiency, while the X reasoning converges very fast.Figure 3 compares performance under partial training data and accuracy across training epochs.
- Explainability: Attention-based XNMs make intermediate reasoning steps transparent and visually traceable on CLEVR examples.Darker red dots indicate higher attention weights for objects in the reasoning visualizations.
- CLEVR-CoGenT: 72.1% accuracy on Condition B with detected scene graphs reveals that fused visual node embeddings retain the dataset shortcut.With ground-truth scene graphs, XNMs perform perfectly on both conditions.
5. Conclusions
The paper concludes that XNMs enable visual reasoning over scene graphs with strong performance under ground-truth representations and substantially compact models.
- The framework supports scene graphs represented with different detection qualities.The conclusion presents this flexibility as a central property of XNMs.
- XNMs achieve 100% accuracy on CLEVR using ground-truth scene graphs and programs.This result establishes the reported performance upper-bound for XNMs in that setting.
- XNMs achieve 100% accuracy with only 0.22M parameters on CLEVR.The model separates high-level reasoning from low-level visual perception.
- The experiments suggest that visual reasoning benefits from high-quality scene graphs.The authors connect this observation to the practical significance of scene graph research.
A. Implementation Details
The implementation uses fixed embedding dimensions and optimizer settings, with mapping matrices differing between ground-truth and detected settings.
- The CLEVR experiments use 128-dimensional label embeddings and 128-dimensional output features.A multilayer perceptron maps the resulting features to 28 possible answers.
- The classifier maps 128-dimensional features to 28 possible answers using a multilayer perceptron and softmax.
- The GT setting uses concatenated attribute embeddings and fixed matrices to extract object attribute features.The four attribute categories are color, shape, size, and material.
- The Det setting learns mapping matrices, increasing the parameter count from 0.22M in GT to 0.55M.
- A sigmoid attention strategy still achieves 100% accuracy in the CLEVR GT setting.This supplements the label-space softmax attention strategy.
- VQAv2.0 training uses an LSTM question encoder and predicts among the 3000 most frequent training answers.The question embedding is fused with the module-network output for answer classification.
B. Failure Cases of the CLEVR Det Setting
CLEVR Det failures arise from inaccurate coordinates, occluded objects, and inaccurate mask-based object division; graph reasoning makes these errors traceable.
- CLEVR Det failures are classified into inaccurate coordinate detection, occluded objects, and inaccurate object division.
- Inaccurate coordinate detection can assign incorrect spatial relationships to detected objects.
- Occluded objects lack corresponding reasoning dots, producing wrong predictions when those objects are needed.
- Blurring boundaries in mask-based object proposals can merge adjacent objects or create extra objects.These errors can make predicted object counts differ from the ground truth.
- XNMs allow the reasoning steps causing mistakes to be tracked and diagnosed.The failure cases mark the responsible reasoning steps in red boxes.
C. Case Study on the VQAv2.0
The VQAv2.0 case study shows that correct answers can sometimes be produced without the Relate module, while detection errors create distinct reasoning failures.
- The VQAv2.0 example answers a relationship-requiring question without substantially relying on Relate.The model directly focuses on the target region instead.
- The authors attribute this behavior to the simplicity of VQAv2.0 questions.They identify simple question annotations as a dataset shortcoming.
- Inaccurate coordinate detection causes incorrect spatial reasoning in the illustrated failure cases.
- Occluded objects missing from reasoning results can change a correct “Yes” answer into a predicted “No”.
- Merging adjacent objects with identical attributes can make the predicted count lower than the ground-truth count.