Source-linked AI summary
Spatial Memory for Context Reasoning in Object Detection
Xinlei Chen, Abhinav Gupta
TL;DR
Object detection lacks memory for conditioning on previous detections, while existing memory approaches generally omit spatial layout. The paper introduces Spatial Memory Network, which assembles detected instances into a pseudo-image for ConvNet-based context reasoning and reports a 2.2% improvement over baseline Faster RCNN on COCO.
Problem
Object detectors detect instances in parallel and rely on manually designed NMS, while prior memory-based approaches generally do not capture spatial layout for instance-level context reasoning.
Method
Spatial Memory Network assembles object instances into a spatial pseudo-image that another ConvNet processes for object-object context reasoning and memory updates.
Results
2.2% improvement over baseline Faster RCNN on the COCO dataset.
Takeaways & Limitations
The framework provides a generic direction for instance-level reasoning and knowledge-based reasoning on images.
Takeaways & Limitations
The authors report that the best SMN performance remains behind the baseline, with recall identified as the biggest issue.
Abstract
from arXiv · showhide
Modeling instance-level context and object-object relationships is extremely challenging. It requires reasoning about bounding boxes of different classes, locations \etc. Above all, instance-level spatial reasoning inherently requires modeling conditional distributions on previous detections. Unfortunately, our current object detection systems do not have any {\bf memory} to remember what to condition on! The state-of-the-art object detectors still detect all object in parallel followed by non-maximal suppression (NMS). While memory has been used for tasks such as captioning, they mostly use image-level memory cells without capturing the spatial layout. On the other hand, modeling object-object relationships requires {\bf spatial} reasoning -- not only do we need a memory to store the spatial layout, but also a effective reasoning module to extract spatial patterns. This paper presents a conceptually simple yet powerful solution -- Spatial Memory Network (SMN), to model the instance-level context efficiently and effectively. Our spatial memory essentially assembles object instances back into a pseudo "image" representation that is easy to be fed into another ConvNet for object-object context reasoning. This leads to a new sequential reasoning architecture where image and memory are processed in parallel to obtain detections which update the memory again. We show our SMN direction is promising as it provides 2.2\% improvement over baseline Faster RCNN on the COCO dataset so far.
1. Introduction
Object detection systems capture image-level context but lack memory for spatial, instance-level reasoning over previous detections. Spatial Memory Network addresses this gap by encoding detected instances as a pseudo-image for ConvNet-based context reasoning and reports benefits on COCO.
- Context can improve visual recognition, but recent detectors have had little success modeling object-object relationships at the instance level.
- Instance-level reasoning must handle relationships among bounding-box groups differing in class, location, scale, and aspect ratio.
- Current detectors detect objects in parallel and apply NMS, lacking memory for conditioning spatial reasoning on previous detections.
- Spatial Memory Network assembles object instances into a pseudo-image that another ConvNet processes for object-object context reasoning.
- The approach demonstrates learned de-duplication as an alternative to manually thresholded NMS and shows context-based reasoning benefits on COCO.
- The spatial memory framework is presented as a general representation for spatial and semantic interactions among groups of objects.
2. Related Work
Related work covers sequential detection, iterative box refinement, reinforcement-learning-based sequence selection, and spatial memory approaches. SMN differs by targeting context for detecting other objects from current detections.
- Sequential region-proposal methods replace exhaustive sliding-window search with smarter region-selection processes.
- Iterative bounding-box refinement improves localization of the original instance, whereas SMN focuses on detecting other objects given current detections.
- Deep reinforcement learning has mainly addressed box refinement or knowledge-assisted detection because full supervision is difficult in high-dimensional action spaces.
- Prior vision research has used higher-dimensional memory for spatial attention, NMS bypassing, planning, and navigation, while this work extends such efforts toward generic multi-class object detection.
3. Background: Faster RCNN
Faster R-CNN proposes candidate regions and classifies them with pooled convolutional features, while NMS removes overlapping duplicates during proposal and final-detection stages.
- SMN is agnostic to the base detector and is demonstrated on Faster R-CNN, which predicts and classifies Regions of Interest.
- The region proposal network uses convolutional layers and anchor boxes across scales and aspect ratios to generate candidate regions.
- The proposal count is K≈h′×w′×s×r^3, with k≪K regions selected for second-stage classification.
- Faster R-CNN applies RoI pooling to arbitrary rectangular proposals before fully connected classification and bounding-box regression.
- Non-maximal suppression prunes overlapping proposals before region classification and removes duplicate final detections in a per-class manner.
4. Spatial Memory Network
The Spatial Memory Network reformulates object detection as sequential conditional inference, storing prior detections in a two-dimensional memory and using that memory for context reasoning. Its pipeline updates memory iteratively and combines memory-derived predictions with Faster R-CNN outputs.
- Formulation: Object instances are jointly modeled through sequential conditional inference rather than independent region detections.The formulation introduces prior detections O_0:n−1 and conditions each new object on them.
- Memory iterations: The spatial memory S_n−1 stores previous detections and is updated after each detected instance.Detection features are written into corresponding memory cells, after which the context model reads the memory for subsequent predictions.
- Memory representation: A two-dimensional memory preserves spatial layout and provides image-like inputs for ConvNet-based context reasoning.The memory uses a fixed 20×20×256 representation resized to the incoming conv5 3 feature-map size.
- Memory indexing and writing: Memory updates combine spatial conv5 3 features with semantic detection scores, using RoI pooling and additional convolutions to merge them.The write process is divided into what features to insert and how to update memory; a convolutional GRU performs the update.
- Context reasoning and outputs: A context network extracts spatial patterns from memory, and its scores are fused with Faster R-CNN scores for region proposals and classification.The context model uses a five-layer all-convolutional network; later iterations augment the original detector predictions with memory predictions.
- Region selection: The next memory update uses the most confident foreground object box under a greedy selection strategy.The paper notes that more advanced region-selection models could exploit which objects are most useful as context, but does not use them here.
5. Training the Spatial Memory
SMN training must learn both sequential context reasoning and de-duplication, which creates conflicting training signals. The authors stabilize learning through gradient control, staged training, specialized sampling, and curriculum learning.
- Learning De-duplication: SMN is trained with back-propagation through time, but conditional detection introduces a difficult de-duplication problem.The network must learn when a detected instance should no longer be detected.
- Learning De-duplication: Directly fusing memory and FRCNN beliefs causes conflicting perception and de-duplication signals that prevent convergence.The contradiction propagates through the network because the memory signal is initially weak and unreliable.
- Learning De-duplication: Stopping gradients to FRCNN from the second iteration allows the network to converge by blocking the adversarial de-duplication signal.The authors also exclude memory-related weights from the first iteration so prediction changes caused by memory can be read directly.
- Training Strategy: Training includes flipped-label regions so examples affected by previous detections are not buried among negative samples.These regions form a distinct training category introduced by spatial-memory de-duplication.
- Training Strategy: RoI sampling improves robustness because sampling k≪K regions can admit alternative confident boxes and reduce over-fitting.The most confident RoI is not guaranteed to be selected when updating memory.
- Training Strategy: The model can be trained in two stages, first optimizing FRCNN independently and then learning spatial-memory reasoning on top of fixed perception features.Curriculum learning bootstraps longer unrolls from shorter ones, letting later iterations focus on object-object relationships.
6. Experimental Results
Experiments evaluate SMN on COCO and VOC, including comparisons with FRCNN and a parameter-matched MLP baseline. The final system improves over baseline FRCNN, while ablations expose recall limitations and different benefits across object sizes.
- Experimental Setup: The experiments use VOC 2007 and COCO, reporting mAP, AP, and AR with dataset-specific evaluation toolkits.COCO uses trainval35k for training and minival for evaluation; VOC uses trainval and test splits.
- Experimental Setup: The sequential setting makes NMS errors especially costly because one missed detection can negatively affect subsequent detections.Because of GPU memory limits, the implementation uses the top 5k regions and unrolls SMN for N=10 iterations.
- SMN for Hard Examples: The hard-example strategy uses standard FRCNN for the first N1=50 detections and SMN for the later N2=10 detections on COCO.Memory is updated throughout, but only later iterations use it as a context provider for harder examples.
- SMN for Hard Examples: SMN is 2.2% better than baseline FRCNN on the final system comparison.The authors attribute the gain to finding hard examples, including some large objects that benefit from improved de-duplication.
- Ablative Analysis: Ablations show that flipped-region sampling helps more on small objects, while multi-task learning helps more on bigger objects.Both strategies improve performance, but their effects differ by object size.
- Ablative Analysis: The best ablative SMN result remains behind baseline, with COCO AR indicating recall as the main issue.The authors therefore conduct additional investigations using SoftMax, HardMax, and non-aggressive NMS.
7. Conclusion
The paper introduces Spatial Memory Network (SMN) as a framework for efficient instance-level context reasoning in object detection. SMN assembles object instances into a pseudo-image for ConvNet-based object-object reasoning and improves baseline Faster R-CNN by 2.2% on COCO.
- SMN models instance-level context by assembling object instances into a pseudo-image representation.The representation can be fed into another ConvNet to extract context information and reason about object-object relationships.
- SMN improves baseline Faster R-CNN by 2.2% on the COCO dataset.
- The framework is presented as a first step toward instance-level reasoning in object detection with ConvNets.
A1. More Qualitative Results
Qualitative examples show SMN using contextual detections and spatial relationships to increase confidence or recover objects that NMS might suppress, while also exhibiting category-confusion failures.
- SMN increases skis confidence using the detected person and their relative location.
- SMN increases tennis-racket confidence despite motion blur by using the person and her pose.
- SMN improves backpack and occluded-sheep detections through contextual and overlap-aware reasoning.For the sheep, SMN goes beyond overlapping NMS reasoning that would prevent the back sheep from being detected.
- SMN fails when potatoes are mistaken for pizza and when a baby is suppressed given the person holding it.
A2. Category-wise Ablative Analysis on VOC
The VOC ablations examine category-wise behavior, SoftMax versus HardMax selection, region sampling, and comparisons with baseline methods. SoftMax improves recall in SMN, while overall VOC results are near baseline and context gains are limited.
- Category-wise analysis: VOC mAP remains below baseline in the ablation: 64.6% versus 65.8% for N=5 and 67.8% versus 70.3% for N=10.
- Category-wise analysis: SoftMax improves SMN recall over HardMax, while distinctive categories such as person remain nearly unchanged.Confusable categories such as cow and sheep receive consistent improvements, and overall SoftMax outperforms HardMax in most cases.
- Region sampling: SMN is more vulnerable than standard detection to aggressive region selection because one missed detection can affect subsequent sequential predictions.
- Region sampling: SMN achieves on-par N=10 or better N=5 mAP despite receiving fewer sequential candidate-selection opportunities than the baseline.The baseline can use more RoIs, creating a subtle testing advantage under HardMax.
- Final comparison: On VOC, SMN is on-par with the baseline and MLP at approximately 71% mAP.The paper attributes the limited context opportunity to VOC having less remaining detection difficulty and training data compared with COCO.
A3. Final Results on VOC
The final VOC evaluation compares Faster R-CNN, an MLP context model, and SMN using SoftMax testing and top 5k RoIs, with a memory-aware evaluation procedure.
- The final VOC comparison evaluates baseline Faster R-CNN, a five-layer ConvNet MLP context model, and SMN.
- All compared methods use SoftMax and top 5k RoIs during final evaluation.Because of memory limitations, the evaluation first uses NMS to find easy examples, stores them in spatial memory, and then predicts with SMN.