PaperlayerView exact revision on arXiv

Source-linked AI summary

Relation Networks for Object Detection

Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, Yichen Wei

arXiv:1711.11575v2cs.CV

TL;DR

Deep-learning detectors largely recognize objects individually, leaving limited evidence that learned object relations improve detection. This paper introduces a lightweight, in-place attention-based relation module that jointly uses appearance and geometry, and shows consistent gains in recognition and duplicate removal, enabling end-to-end detection.

  • Problem

    Deep-learning object detectors generally recognize instances separately, with limited evidence that modeling object relations improves detection learning.

  • Method

    The paper adapts attention into an object relation module that jointly processes object appearance and relative geometry.

  • Results

    The module consistently improves instance recognition and duplicate removal across several state-of-the-art detection architectures, yielding an end-to-end detector.

  • Takeaways & Limitations

    The results verify the efficacy of modeling object relations in CNN-based detection without additional supervision.

  • Takeaways & Limitations

    The authors state that their understanding of what stacked relation modules learn remains preliminary and leave further analysis for future work.

Abstract

from arXiv · show

Although it is well believed for years that modeling relations between objects would help object recognition, there has not been evidence that the idea is working in the deep learning era. All state-of-the-art object detection systems still rely on recognizing object instances individually, without exploiting their relations during learning. This work proposes an object relation module. It processes a set of objects simultaneously through interaction between their appearance feature and geometry, thus allowing modeling of their relations. It is lightweight and in-place. It does not require additional supervision and is easy to embed in existing networks. It is shown effective on improving object recognition and duplicate removal steps in the modern object detection pipeline. It verifies the efficacy of modeling object relations in CNN based detection. It gives rise to the first fully end-to-end object detector.

1. Introduction

The paper addresses the lack of evidence for learning object relations in deep-learning detection by introducing an attention-based object relation module. The module jointly reasons over object appearance and geometry, improving recognition and duplicate removal while fitting existing detectors.

  • Deep-learning detectors generally recognize region proposals independently and use hand-crafted non-maximum suppression to remove duplicates.
  • Object relations remain difficult to model because objects vary in location, scale, category, and number across images.
  • The proposed object relation module adapts attention to objects by combining appearance-based and relative geometric weights.The geometric component models spatial relationships and is translation invariant.
  • The module accepts variable-sized object sets, runs in parallel, is fully differentiable, and preserves input-output dimensions.These properties make it a flexible building block for existing architectures.
  • Applied to several state-of-the-art architectures, the module consistently improves instance recognition and duplicate removal.Replacing traditional duplicate removal with a relation network yields an end-to-end detector.
  • The approach processes and reasons over objects simultaneously instead of recognizing each object individually.The module is presented as a new relational dimension for CNN-based detection.

2. Related Works

The related work spans relational post-processing, sequential reasoning, human-object modeling, duplicate removal, and attention mechanisms. The paper positions its module as a parallel, general, lightweight alternative that incorporates geometric relations into object detection.

  • Object relation in post-processing: Earlier object-relation methods mainly used relations during post-processing to re-score detected objects.Examples include class co-occurrence and models using object position and size.
  • Object relation in post-processing: Deep-learning-era methods did not establish that object relations improve learning in deep convolutional networks.Large receptive fields may already incorporate contextual information implicitly.
  • Sequential relation modeling: Sequential relation models use earlier detections to help find later objects but require sophisticated training and lack evidence of improving state-of-the-art detectors.
  • Sequential relation modeling: The proposed approach processes multiple objects in parallel, fitting naturally into and improving modern object detectors.
  • Human centered scenarios: Unlike human-object relation methods, the module models general object-object relations without requiring additional relation annotations.
  • Duplicate removal: For duplicate removal, the paper replaces hand-crafted NMS with a simpler relation network that has small overhead and surpasses SoftNMS.The authors report that this makes end-to-end object detection learning feasible and effective.
  • Attention modules in NLP and physical system modeling: The module extends attention modeling with a translation-invariant geometric weight for spatial relationships between objects.

3. Object Relation Module

The object relation module applies attention to object appearance features while incorporating relative geometry into the relation weights. It aggregates information across a variable-sized object set and remains lightweight, differentiable, and usable in-place within existing detection architectures.

  • The module represents each object with a 4-dimensional bounding-box geometry feature and a task-dependent appearance feature.
  • For each object, relation features are computed by aggregating appearance features from other objects using learned attention weights.The weights reflect appearance similarity and geometric relationships between object pairs.
  • The module extends basic attention by combining appearance-based weights with a learned geometric weight based on embedded relative geometry.The relative geometry embedding uses translation- and scale-invariant features, while zero trimming restricts relations to selected geometric configurations.
  • It aggregates Nr relation features and augments each input appearance feature through addition, with concatenation used to combine multiple relation features.
  • With typical values Nr = 16, dk = 64, and dg = 64, the overall computation overhead is low in modern object detectors.The stated complexity is O(Comp.) = Ndf(2Nrdk+df)+N^2Nr(dg+dk+df/Nr+1).
  • The module accepts a variable number of inputs, runs in parallel, is fully differentiable, and preserves input and output dimensions for in-place use.It can therefore serve as a basic building block within different network architectures.

4. Relation Networks For Object Detection

The paper embeds object relation modules into the final stages of region-based detection to jointly improve instance recognition and learn duplicate removal. Jointly training these stages further boosts accuracy and produces an end-to-end detector.

  • Detection pipeline: The region-based pipeline extracts full-image features, proposal-specific regional features, per-proposal recognition and regression outputs, then removes duplicates with NMS.NMS is heuristic, manually designed, and prevents end-to-end learning.
  • End-to-end training: Joint training of instance recognition and duplicate removal improves recognition accuracy and gives rise to the first end-to-end general object detection system.Despite apparently conflicting objectives, the multiplicative final score makes the two stages complementary in the reported experiments.
  • Instance recognition: Relation modules enhance the shared 2fc head by transforming all 1024-d proposal features while preserving their dimensionality.They can be inserted after either fully connected layer and repeated an arbitrary number of times.
  • Instance recognition: Adding relation modules improves instance recognition accuracy, as shown through ablation studies.The modules jointly reason over proposals rather than processing each proposal independently.
  • Duplicate removal: The duplicate-removal network classifies each detected object as correct or duplicate using its appearance feature, classification score, and bounding box.Its binary output s1 is multiplied by the original score s0 to form the final classification score.
  • Duplicate removal: Learning duplicate removal can adapt selection criteria to evaluation needs, unlike NMS, whose threshold choice favors the matching mAP threshold.Using η = 0.5 in learning produces best mAP@0.5 but not mAP@0.75; a large η is suggested when high localization accuracy is desired.

5. Experiments

Experiments show that relation modules improve instance recognition and duplicate removal across object-detection settings, with gains beyond simply increasing network capacity. The modules add relatively small computational overhead and support end-to-end training.

  • Relation-module ablations: +2.3 mAP is achieved at Nr = 16, with further relation modules providing up to +3.2 mAP gain.Using more relations steadily improves accuracy until saturation at Nr = 16; using more modules steadily improves accuracy but also increases complexity.
  • Relation-module ablations: +2.3 mAP is obtained with the relation-module head, compared with +0.1 mAP for a wider 2fc head and -0.6 mAP for a deeper 3fc head.Residual blocks yield only +0.3 mAP, while global context yields no improvement under comparable-complexity comparisons.
  • Complexity: Each relation module adds less than 2% overhead for Faster R-CNN and DCN and about 8% for FPN.With Nr = 16, a module has about 3 million parameters and 1.2 billion FLOPs.
  • Duplicate removal: 30.3 mAP is obtained for duplicate removal with the full input features, while removing the rank feature reduces mAP to 26.6.Removing geometric features reduces mAP to 28.1, and replacing the rank with the class score reduces mAP to 28.3.
  • Duplicate removal: The relation-based duplicate-removal method achieves better final mAP than NMS and SoftNMS.Its parameter η performs best for mAP50 at 0.5, for mAP75 at 0.75, and for mAP over the range [0.5, 0.9].
  • End-to-end detection: +0.5 mAP is obtained by end-to-end learning compared with separate training of instance recognition and duplicate removal.On stronger systems, the method improves Faster R-CNN with ResNet-101 by 2.5 mAP before the additional 0.5 mAP gain from end-to-end duplicate removal.

6. Conclusions

The conclusions attribute the improvements to information between objects that is absent when objects are learned individually. However, the authors state that their understanding of what stacked relation modules learn remains preliminary.

  • Conclusions: Comprehensive ablations suggest that relation modules learn information between objects missing from individual-object learning.The conclusion frames this as the interpretation of the observed ablation results.
  • Conclusions: Examples suggest that overlapping bicycle detections contribute to a central object and that a person contributes to a glove.These examples are presented as intuitive indications of relation-module behavior.
  • Conclusions: The authors describe their understanding of relation-module behavior, especially with multiple stacked modules, as preliminary and leave it for future work.The investigation covers only the relation module in the {r1, r2} = {1, 0} head.

A1. Training Details

The experiments use distinct training configurations for Faster R-CNN/DCN and FPN, with specified image sizes, proposal counts, optimization settings, and training schedules. Instance-recognition training uses all proposals rather than sampled subsets.

  • Faster R-CNN and DCN: Faster R-CNN and DCN training resizes images to a 600-pixel shorter side and uses 300 region proposals.Region proposal and instance-recognition networks are jointly trained for approximately 450k iterations.
  • FPN: FPN training resizes images to an 800-pixel shorter side and uses 1000 region proposals for training and testing.The region proposal network is trained for approximately 170k iterations, while instance recognition and end-to-end training use approximately 340k iterations.
  • Optimization: All training uses SGD on four GPUs with one image per GPU, weight decay of 1×10^-4, and momentum of 0.9.Class-agnostic bounding-box regression is used for efficiency.
  • Proposal usage: All N proposals are used for instance-recognition loss, with similar accuracy to sampled-proposal training.The experiments also consider online hard example mining for stronger baseline performance.
Loading 1711.11575v2…