Source-linked AI summary
Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships
Yong Liu, Ruiping Wang, Shiguang Shan, Xilin Chen
TL;DR
Object detectors often rely mainly on local visual appearance, limiting their use of scene context and relationships among objects. The paper introduces SIN, a graph-based detector that iteratively infers object states from object, scene, and relationship information. Experiments on PASCAL VOC and MS COCO report improved detection performance, including 76.0% mAP on VOC 2007, 73.1% mAP on VOC 2012, and 23.2% AP on COCO test-dev 2015.
Problem
Existing object detectors focus mainly on local appearance, despite images containing scene context and object relationships that can support recognition.
Method
SIN formulates detection as graph structure inference, using object proposals as nodes, relationships as edges, scene context, and GRU-based message passing to update object states.
Results
SIN improves detection across PASCAL VOC and MS COCO, achieving 76.0% mAP on VOC 2007, 73.1% mAP on VOC 2012, and 23.2% AP on COCO test-dev 2015.
Takeaways & Limitations
Scene context is especially useful for categories correlated with their surroundings, while instance-level relationships particularly improve object localization accuracy.
Abstract
from arXiv · showhide
Context is important for accurate visual recognition. In this work we propose an object detection algorithm that not only considers object visual appearance, but also makes use of two kinds of context including scene contextual information and object relationships within a single image. Therefore, object detection is regarded as both a cognition problem and a reasoning problem when leveraging these structured information. Specifically, this paper formulates object detection as a problem of graph structure inference, where given an image the objects are treated as nodes in a graph and relationships between the objects are modeled as edges in such graph. To this end, we present a so-called Structure Inference Network (SIN), a detector that incorporates into a typical detection framework (e.g. Faster R-CNN) with a graphical model which aims to infer object state. Comprehensive experiments on PASCAL VOC and MS COCO datasets indicate that scene context and object relationships truly improve the performance of object detection with more desirable and reasonable outputs.
1. Introduction
The paper argues that object detection should use scene context and object relationships alongside local visual appearance. It formulates detection as graph-based reasoning and introduces SIN to infer object states from these complementary signals.
- Motivation: Modern detectors mainly classify candidate boxes using local region appearance, while images also contain scene context and object relationships.The paper identifies ignoring these contextual cues as a constraint on detection accuracy.
- Motivation: Faster R-CNN mislabels boats as cars in a river scene because it focuses on visual appearance rather than scene information.The example illustrates how scene context can disambiguate visually similar objects.
- Motivation: Faster R-CNN misses a mouse despite detecting a laptop and person, although mouse-laptop co-occurrence and relative position provide useful cues.Instance-level relationships could support detection of additional objects in the image.
- Motivation: Scene-level context and instance-level relationships capture complementary information and can be jointly used to improve detection.Prior studies explored these contextual types separately, motivating their combination.
- Proposed formulation: SIN models objects as graph nodes and relationships as edges, with scene context guiding message exchange to update object states.The resulting object states determine categories and refine locations.
- Proposed formulation: SIN uses GRU-based memory cells to encode scene and object messages into object states and is not restricted to a specific detection framework.Object representations initialize the GRU, while contextual messages update the states.
2. Related Work
Prior detection methods include proposal-based and proposal-free CNN detectors, while contextual and graphical-model approaches extend recognition beyond local appearance. SIN builds on these directions by combining scene, object, and relationship information in a detection framework.
- Object detection: Modern CNN detectors comprise two-stage region-proposal methods and one-stage proposal-free methods.Two-stage detectors first generate candidate boxes and then classify them, whereas one-stage methods target real-time detection.
- Contextual information: Context-aware detection methods have modeled local surroundings, global scene context, and penalties for objects appearing in irrelevant scenes.These approaches expand recognition beyond the candidate region itself.
- Structure inference: Structured prediction research combines deep networks with graphical models to exploit relations among scenes, objects, attributes, activities, and interactions.Related applications include group activity recognition and structured scene-graph generation.
- Structure inference: SIN differs by explicitly propagating information among object proposals and the whole scene within a detector framework.Its framework extracts ROI nodes, scene features, and pairwise ROI edges before iterative structure inference.
3. Method
SIN extends object detection with graph-based message passing that integrates scene context and object relationships into updated object states. Built with GRU memory cells, it iteratively propagates contextual information before predicting categories and bounding-box offsets.
- Graphical Modeling: SIN propagates scene information and object-object relationships through a graph whose nodes are region proposals and whose edges represent relationships.The framework is agnostic to the base detector, demonstrated with Faster R-CNN.
- Graphical Modeling: The detector extracts visual features for each ROI and the whole image, while directed-edge weights encode spatial and visual influence between object proposals.Each relationship is represented as a scalar computed from the paired objects’ spatial and visual features.
- Message Passing: GRU memory cells update object states by selectively retaining or discarding prior information while incorporating scene or neighboring-object messages.The reset gate can discard irrelevant state information, and the update gate controls how much prior state carries forward.
- Message Passing: Scene GRUs and edge GRUs separately encode scene and inter-object messages, then their outputs are combined into the updated node representation.Scene GRUs use node features as initial states and scene features as inputs; edge GRUs aggregate messages from other objects.
- Structure Inference: Mean-pooling was empirically most effective for integrating incoming object messages compared with max-pooling and concatenation.The study reports this comparison in its current experiments.
- Structure Inference: The model iterates state and relationship updates, and the final integrated node representations predict object categories and bounding-box offsets.Subsequent iterations reuse updated node states and recomputed object-object messages.
4. Results
SIN was evaluated on PASCAL VOC and MS COCO using a Faster R-CNN baseline, achieving higher reported performance on both datasets.
- Evaluation setup: The evaluation covered PASCAL VOC and MS COCO, with Faster R-CNN trained as the baseline.The setup used a VGG-16 model pretrained on ImageNet and selected 128 object proposals with NMS.
- PASCAL VOC: 76.0% mAP was achieved on VOC 2007, while VOC 2012 test reached 73.1% mAP.The experiments used VOC trainval data and evaluated on the corresponding test sets.
- PASCAL VOC: The method was reported as better than ION on the VOC 2007 test set.ION is described as a multi-scale network with explicit context modeling through a recurrent network.
- MS COCO: SIN achieved 23.2% on the MS COCO test-dev 2015 evaluation.COCO AP averages performance across IoU thresholds from 0.5 to 0.95, emphasizing localization more than the VOC metric.
5. Design Evaluation
The evaluation separates scene context and object relationships, then studies their fusion and iterative updating in SIN. Both modules improve detection, with scene context helping context-correlated and difficult objects, while Edge improves localization and reduces redundant detections.
- Scene Module: Scene achieves a 70.23% mAP on VOC, with boat average precision increasing by more than 6% over the baseline.The strongest gains occur for categories highly correlated with scene context, such as planes, birds, and boats.
- Scene Module: Scene is more robust than the baseline against occlusion, truncation, area size, and part visibility, improving especially extra-small bird, boat, and cat instances.Small-object AP S on COCO also improves compared with the baseline.
- Scene Module: Scene context corrects some vague boat detections and recovers a missed chair, but can misclassify rare aeroplanes appearing on rivers.The failure case motivates balancing global scene context between common and uncommon situations.
- Edge Module: Edge largely decreases localization errors, including for aeroplane and bus, and improves AP 70 on COCO.The analysis attributes this to more accurate localization compared with the baseline.
- Edge Module: Edge reduces redundant overlapping detections by homogenizing related nodes, producing higher accuracy and visually cleaner results.The module addresses multiple high-scoring boxes with similar labels generated from one candidate region.
- Ensemble: Mean-pooling is the best fusion strategy, while two update steps give the highest performance and additional steps degrade it as noisy messages spread through the graph.The experiments compare max-pooling, mean-pooling, concatenation, and different recurrent update counts.
- Ensemble: SIN attains higher precision than the baseline with almost the same recall, although relationship constraints may limit recall for rare scene-object configurations.At similar positive-instance recall, SIN produces fewer and more accurate detections.
6. Conclusion
The paper concludes that SIN jointly uses scene context and object relationships for detection. Across VOC and COCO evaluations, scene context helps context-correlated categories while instance relationships particularly improve localization, with rare failures in uncommon situations.
- Conclusion: SIN jointly incorporates scene context and object relationships through a structure inference network for object detection.The method is evaluated on VOC and COCO datasets.
- Conclusion: Scene-level context is useful for categories strongly correlated with scenes, although rare failures can occur in uncommon situations.The conclusion specifically notes uncommon scene-object configurations as a boundary for scene-based reasoning.
- Conclusion: Instance-level relationships play an important role in detection and particularly improve object localization accuracy.The paper reports this conclusion from evaluations spanning VOC and COCO.