Source-linked AI summary
Iterative Visual Reasoning Beyond Convolutions
Xinlei Chen, Li-Jia Li, Li Fei-Fei, Abhinav Gupta
TL;DR
The paper targets the limited ability of convolution-based recognition systems to reason over spatial and semantic relationships. It introduces iterative local spatial memory and global graph reasoning, achieving an 8.4% absolute ADE improvement over plain ConvNets in per-class average precision and showing resilience to missing regions.
Problem
Convolution-based recognition systems lack context reasoning over spatial and semantic relationships, while relationship examples may be insufficient for learning semantic reasoning from data.
Method
The framework iteratively combines a local spatial-memory module with a global graph module containing knowledge, region, and assignment graphs, with parallel memory updates.
Results
8.4% absolute improvement on ADE over plain ConvNets was achieved using per-class average precision, and the framework was resilient to missing regions.
Takeaways & Limitations
Structured spatial and semantic relationships can be incorporated into iterative visual recognition beyond convolutional processing.
Abstract
from arXiv · showhide
We present a novel framework for iterative visual reasoning. Our framework goes beyond current recognition systems that lack the capability to reason beyond stack of convolutions. The framework consists of two core modules: a local module that uses spatial memory to store previous beliefs with parallel updates; and a global graph-reasoning module. Our graph module has three components: a) a knowledge graph where we represent classes as nodes and build edges to encode different types of semantic relationships between them; b) a region graph of the current image where regions in the image are nodes and spatial relationships between these regions are edges; c) an assignment graph that assigns regions to classes. Both the local module and the global module roll-out iteratively and cross-feed predictions to each other to refine estimates. The final predictions are made by combining the best of both modules with an attention mechanism. We show strong performance over plain ConvNets, \eg achieving an $8.4\%$ absolute improvement on ADE measured by per-class average precision. Analysis also shows that the framework is resilient to missing regions for reasoning.
1. Introduction
The paper addresses the limited spatial and semantic reasoning of convolution-based recognition systems by introducing an iterative framework that combines local memory with global graph reasoning. It reports stronger performance than plain ConvNets, including an 8.4% absolute improvement on ADE per-class average precision.
- Motivation: Convolution-based systems advance recognition but lack context reasoning over spatial and semantic relationships.The paper frames incorporating both relationship types as a critical question for next-generation vision systems.
- Reasoning examples: Spatial and semantic reasoning can support recognition from context, such as inferring an occluded window, recognizing a school bus from bus examples, or linking a person to a car.These examples motivate reasoning beyond isolated convolutional features.
- Limitations of prior approaches: Existing top-down and memory-based approaches primarily perform local pixel-level reasoning and cannot directly communicate information between distant regions.These approaches also assume sufficient relationship examples during training.
- Framework: The proposed framework combines a local spatial-memory module with a global graph module that represents class semantics, image-region spatial relations, and region-to-class assignments.Both modules iteratively roll out and cross-feed predictions to refine estimates.
- Results: 8.4% absolute improvement on ADE was achieved over plain ConvNets using per-class average precision.Simply making the network deeper improved performance by only ∼1%.
2. Related Work
The related work distinguishes between collecting structured visual knowledge and using that knowledge for reasoning in end tasks. The proposed framework belongs to the latter direction and applies visual knowledge bases to recognition.
- Visual Knowledge Base: Large-scale visual knowledge includes labeled object and scene instances as well as structured relationships accumulated from datasets and the web.Prior efforts include crowdsourced labels and automatically collected relationships or ngrams.
- Visual Knowledge Base: Earlier work often focused on building knowledge bases, whereas related approaches applied visual knowledge to affordances, image classification, or question answering.The paper positions its framework as a more general approach along this application-oriented line.
3. Reasoning Framework
The framework combines a local spatial-memory module with a global graph module to iteratively refine visual predictions. Local and global beliefs cross-feed across iterations, and attention fuses predictions from both modules and the plain ConvNet.
- Framework overview: The framework adds local and global reasoning modules alongside a plain ConvNet, with iterative roll-outs and cross-fed beliefs.The local module uses spatial memory, while the global module reasons over regions and classes represented as graph nodes.
- Local module: The local module stores previous beliefs in spatial memory and updates overlapping regions in parallel using weighted averaging.The memory preserves spatial layout, and each cell combines contributions according to how much each region covers it.
- Local module: The local reasoning ConvNet operates on spatial memory, preserving two-dimensional structure and location information for spatial relationships.Its receptive fields build connections within local windows of the memory.
- Global module: The global module represents image regions and classes as graph nodes, enabling direct spatial communication beyond local receptive fields and semantic reasoning through class relationships.Its edge groups encode region-to-region spatial relations, region-to-class assignments, and class-to-class semantic relations from knowledge graphs.
- Global module: The global module combines spatial and semantic reasoning paths, then stacks graph operations with residual connections to produce improved region features.The semantic path maps regions to classes, propagates through class relationships, and returns information to regions.
- Global module: Graph message passing transforms node features separately for each edge type, applies the corresponding adjacency matrix, and accumulates the resulting messages.The global module uses these operations to propagate information across multiple relationship types.
- Prediction fusion: The final prediction combines all plain, local, and global outputs using learned attention values that represent relative confidence across iterations and modules.With I roll-outs, the framework produces N=2I+1 predictions before attention-based fusion.
4. Experiments
Experiments evaluate iterative visual reasoning on ADE, Visual Genome, and COCO using region classification, ablations, and missing-region analysis. The combined reasoning modules outperform baselines on ADE and retain gains when regions are missing, though performance eventually declines when very few regions remain.
- Experimental setup: The framework is evaluated on ADE, Visual Genome, and COCO, with ADE and Visual Genome providing densely labeled, large-vocabulary settings suited to context and knowledge-graph reasoning.COCO is used only for analysis without the knowledge graph and without a test split because it is detection-oriented, has 80 mutually exclusive classes, and covers fewer labeled pixels.
- Experimental setup: Region classification assigns labels to designated ground-truth bounding boxes, avoiding ambiguous evaluation from missing labels while retaining stuff, object-instance, and relationship categories.ADE masks are converted to boxes, with separate boxes for object instances, parts, and part-of-parts; VG and COCO use their provided boxes.
- Main results: The framework’s gains are larger for per-class than per-instance metrics, indicating greater help for rare classes, while Visual Genome shows a similar but less significant trend.The authors associate the Visual Genome discrepancy between metrics with noisier labels and missing ground-truth annotations.
- Analysis: Ablations show that spatial memory, global reasoning, cross-feeding, and iterative updates materially support performance, whereas removing the global memory does not hurt much.Replacing spatial memory with direct conv4 features drops performance almost to baseline; removing global reasoning causes a steep drop, and hard-example re-weighting adds around 0.5%.
- Analysis: With an IoU threshold of 0.5 retaining 78.1% of ground-truth boxes, the model still beats the baseline by 2.4% in post and 2.2% in pre missing-region settings.Performance degrades smoothly as regions are dropped, but reasoning hurts at δ=.9 when recall falls to 3.9%.
5. Conclusion
The framework performs iterative visual reasoning beyond convolutions by encoding spatial and semantic relationships in a graph. It improves recognition over plain ConvNets and remains resilient to missing regions.
- 8.4% absolute gain on ADE and 3.7% on COCO over plain ConvNets.
- The framework encodes spatial and semantic relationships between regions and classes in a graph.
- The reasoning framework is resilient to missing regions caused by current region proposal approaches.