Source-linked AI summary
DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
Siyuan Qiao, Liang-Chieh Chen, Alan Yuille
TL;DR
Object detectors have benefited from looking and thinking twice, but this paper explores that mechanism within the backbone design. DetectoRS combines recursive FPN feedback with switchable atrous convolution, achieving state-of-the-art results across COCO detection, instance segmentation, and panoptic segmentation.
Problem
The paper explores how the looking-and-thinking-twice mechanism can be incorporated into the backbone design for object detection.
Method
DetectoRS combines Recursive Feature Pyramid feedback connections at the macro level with Switchable Atrous Convolution using different atrous rates at the micro level.
Results
DetectoRS sets state-of-the-art results on COCO for object detection, instance segmentation, and panoptic segmentation, including 55.7% box AP, 48.5% mask AP, and 50.0% PQ.
Takeaways & Limitations
The combined design improves HTC across object detection and instance segmentation while maintaining inference speed in the reported experiments.
Abstract
from arXiv · showhide
Many modern object detectors demonstrate outstanding performances by using the mechanism of looking and thinking twice. In this paper, we explore this mechanism in the backbone design for object detection. At the macro level, we propose Recursive Feature Pyramid, which incorporates extra feedback connections from Feature Pyramid Networks into the bottom-up backbone layers. At the micro level, we propose Switchable Atrous Convolution, which convolves the features with different atrous rates and gathers the results using switch functions. Combining them results in DetectoRS, which significantly improves the performances of object detection. On COCO test-dev, DetectoRS achieves state-of-the-art 55.7% box AP for object detection, 48.5% mask AP for instance segmentation, and 50.0% PQ for panoptic segmentation. The code is made publicly available.
1. Introduction
DetectoRS brings the “looking and thinking twice” design philosophy into the backbone through macro-level feedback and micro-level switchable convolutions, improving detection and segmentation results while maintaining inference speed.
- Motivation: Modern detectors motivate applying feedback and repeated processing directly to backbone design for object detection.The motivation draws on human visual perception and multi-stage detector designs that selectively process increasingly informative features.
- Macro Design: Recursive Feature Pyramid: Recursive Feature Pyramid adds feedback connections from top-down FPN layers to bottom-up backbone layers, enabling repeated image processing.The recursive structure can be unrolled into a sequential backbone that looks at images twice or more.
- Micro Design: Switchable Atrous Convolution: Switchable Atrous Convolution applies different atrous rates to the same input feature and combines the outputs with spatially dependent switches.SAC converts standard 3x3 bottom-up backbone convolutions and supports pretrained standard convolutional networks through weight locking.
- Results: 55.7% box AP and 48.5% mask AP are achieved by DetectoRS with ResNeXt-101-64x4d on COCO test-dev.The reported results are state-of-the-art for object detection and instance segmentation.
- Results: 50.0% PQ is achieved for panoptic segmentation, while the paper reports maintained inference speed.The panoptic result combines DetectoRS with stuff prediction from DeepLabv3+ using Wide-ResNet-41.
2. Related Works
Related work positions DetectoRS among one-stage and multi-stage detectors, repeated feature-pyramid designs, and atrous-convolution methods that enlarge receptive fields.
- Object Detection: Object detectors are commonly divided into one-stage and multi-stage methods, with multi-stage detectors usually more flexible and accurate but more complex.The paper uses multi-stage HTC as its baseline while comparing against both categories.
- Multi-Scale Features: NAS-FPN and Auto-FPN search for FPN structures, while EfficientDet repeats a simple BiFPN layer.These methods provide context for alternative approaches to multi-scale feature aggregation.
- Multi-Scale Features: Recursive Feature Pyramid repeatedly passes through the bottom-up backbone to enrich FPN representations, differing from repeated simple BiFPN layers.The paper also incorporates ASPP into FPN to enrich features.
3. Recursive Feature Pyramid
Recursive Feature Pyramid (RFP) extends FPN with feedback from top-down pyramid features into the bottom-up backbone, then unrolls this recursion across sequential steps. It transforms and fuses intermediate features to repeatedly enrich representations.
- 3. Recursive Feature Pyramid: The architecture overview distinguishes standard FPN, feedback-enhanced RFP, and its two-step sequential unrolling, while the fusion design uses Sigmoid outputs to combine steps.These views describe the macro-level recursive design and its feature-update mechanism.
- 3. Recursive Feature Pyramid: RFP adds feedback connections from FPN layers into the bottom-up backbone, making the pyramid a recursive operation.The backbone equipped with FPN produces multi-stage feature maps used for detection, while RFP feeds transformed pyramid features back into the backbone.
- 3. Recursive Feature Pyramid: RFP is unrolled into a sequential network with T iterations; the implementation uses different bottom-up blocks and typically sets T = 2.Top-down FPN operations are shared across unrolled steps, while the experiments use different bottom-up operations unless otherwise stated.
- 3. Recursive Feature Pyramid: The feedback enters the first block of each ResNet stage through an additional 1x1 convolutional layer that accepts transformed pyramid features.This layer is initialized with zero weights so pretrained backbone weights can be loaded without initially changing the network's behavior.
- 3.3. ASPP as the Connecting Module: ASPP implements the connecting module R with four parallel branches whose outputs are concatenated to produce a feature matching the input channel size.Three branches use convolution and ReLU, while the fourth uses global average pooling, a 1x1 convolution, ReLU, resizing, and concatenation.
- 3. Recursive Feature Pyramid: A fusion module combines features from consecutive unrolled steps using an attention map produced by convolution followed by Sigmoid.The attention-weighted combination updates the feature used in the next step; the module is applied from steps 2 through T.
4. Switchable Atrous Convolution
Switchable Atrous Convolution (SAC) combines convolutions at different atrous rates with spatially dependent switches, allowing features to adapt to object scale. The design also supports pretrained initialization and adds lightweight global-context modules around the main SAC component.
- Atrous Convolution: SAC enlarges the filter field of view by applying atrous convolution with different rates without increasing parameters or computation.For a k × k filter, rate r gives an effective size k_e = k + (k − 1)(r − 1).
- Switchable Atrous Convolution: SAC applies multiple atrous convolutions to the same input and combines their outputs using spatially dependent switch functions.Each feature-map location can use different switches to control the output.
- Switchable Atrous Convolution: The switch function uses average pooling with a 5x5 kernel followed by a 1x1 convolutional layer, with r set to 3 unless otherwise stated.The switch is input and location dependent, enabling adaptation to different scales.
- Pretrained Initialization: SAC locks one convolution’s weights as w and the other’s as w + ∆w, enabling conversion from pretrained standard-convolution checkpoints.The shared initialization reflects the use of different atrous rates to detect objects at different scales.
- Global Context: Two lightweight global-context modules surround the main SAC component, compressing input features with global average pooling before processing.Their outputs are added back to the main stream rather than used as multiplicative recalibration values.
5. Experiments
Experiments on COCO evaluate DetectoRS through component ablations, training behavior, visualizations, and comparisons across detection, instance segmentation, and panoptic segmentation. RFP and SAC each improve the HTC baseline, while their combination achieves strong results and learned switches align with object scale.
- 5.2. Ablation Studies: DetectoRS combines RFP and SAC to achieve 49% box AP and 42.1% mask AP at 3.9 fps with ResNet-50.Both components improve the HTC baseline by >4% AP without too much speed decrease.
- 5.2. Ablation Studies: RFP and SAC ablations identify the proposed formulations as the best configurations within the explored design space.Increasing RFP unrolling from 2 to 3 further improves box AP by 1.3%, and the ablations support SAC weight locking.
- 5.2. Ablation Studies: RFP more easily finds occluded objects using nearby context, whereas SAC better detects large objects by increasing the field of view as needed.The observed SAC behavior is consistent with its higher APL in the ablation results.
- 5.2. Ablation Studies: RFP, SAC, and DetectoRS accelerate training and converge to lower losses over the 12-epoch comparison.The comparison includes HTC, HTC + RFP, HTC + SAC, and DetectoRS.
- 5.4. Visualizing Learned Switches: The learned SAC switch outputs align with ground-truth object scales, using larger atrous rates more for large objects.Darker visualization intensity indicates greater aggregation from the larger atrous rate.
6. Conclusion
The paper proposes DetectoRS, combining Recursive Feature Pyramid and Switchable Atrous Convolution to implement looking and thinking twice at macro and micro levels. Tested on COCO across three tasks, DetectoRS sets new state-of-the-art results.
- 6. Conclusion: DetectoRS combines Recursive Feature Pyramid and Switchable Atrous Convolution.RFP uses feedback from FPN outputs to the bottom-up backbone, while SAC uses two atrous rates.
- 6. Conclusion: RFP implements thinking twice at the macro level through feedback connections from FPN outputs into each bottom-up backbone stage.
- 6. Conclusion: SAC instantiates looking twice at the micro level by convolving inputs with two different atrous rates.
- 6. Conclusion: DetectoRS sets new state-of-the-art results on COCO object detection, instance segmentation, and panoptic segmentation.