Source-linked AI summary
Revisiting the Sibling Head in Object Detector
Guanglu Song, Yu Liu, Xiaogang Wang
TL;DR
Sibling heads can suffer because classification and localization prefer different spatial evidence, limiting detector performance. The paper introduces TSD, which derives separate task-specific proposals from a shared proposal, and adds a progressive constraint. TSD improves performance by ~3% mAP, with the constraint adding ~1% more mAP across COCO and OpenImage settings.
Problem
Spatial misalignment between classification and localization in sibling heads creates conflicting spatial requirements that limit detection performance.
Method
TSD decouples classification and localization spatially by generating two task-specific proposals from a shared proposal, with progressive constraint further separating their performance.
Results
~3% mAP improvement is reported across backbones and models on MS COCO and Google OpenImage, with progressive constraint providing ~1% more mAP.
Takeaways & Limitations
TSD consistently improves varied backbones and detection frameworks on COCO and OpenImage, while its spatial disentanglement alleviates conflict in the sibling head.
Abstract
from arXiv · showhide
The ``shared head for classification and localization'' (sibling head), firstly denominated in Fast RCNN~\cite{girshick2015fast}, has been leading the fashion of the object detection community in the past five years. This paper provides the observation that the spatial misalignment between the two object functions in the sibling head can considerably hurt the training process, but this misalignment can be resolved by a very simple operator called task-aware spatial disentanglement (TSD). Considering the classification and regression, TSD decouples them from the spatial dimension by generating two disentangled proposals for them, which are estimated by the shared proposal. This is inspired by the natural insight that for one instance, the features in some salient area may have rich information for classification while these around the boundary may be good at bounding box regression. Surprisingly, this simple design can boost all backbones and models on both MS COCO and Google OpenImage consistently by ~3% mAP. Further, we propose a progressive constraint to enlarge the performance margin between the disentangled and the shared proposals, and gain ~1% more mAP. We show the \algname{} breaks through the upper bound of nowadays single-model detector by a large margin (mAP 49.4 with ResNet-101, 51.2 with SENet154), and is the core model of our 1st place solution on the Google OpenImage Challenge 2019.
1. Introduction
The paper identifies spatial misalignment between classification and localization within sibling heads as a performance bottleneck and proposes TSD to disentangle their spatial cues. TSD and a progressive constraint improve detection performance across benchmarks, reaching 49.4 mAP with ResNet-101 and 51.2 with SENet154.
- Motivation: Sibling heads share parameters between classification and localization, creating conflict between their different spatial requirements.Classification may favor salient object regions, whereas localization may favor boundary features.
- Related work: Prior methods reduce or separate shared parameters, but both tasks can still receive features derived from the same proposal.IoU-Net adjusts confidence scores, while Double-Head RCNN adds task-specific branches without fully resolving shared spatial clues.
- Method: TSD generates two task-specific proposals from a shared proposal, allowing classification and localization to adapt their spatial focus independently.The operator targets the spatial dimension rather than only evolving the backbone or feature extractor.
- Results: ~3% mAP improvement is reported across backbones and models on both MS COCO and Google OpenImage with TSD.The paper further introduces progressive constraint to enlarge the performance margin between TSD and the classical sibling head.
- Results: ~1% more mAP is gained by adding the progressive constraint to TSD.The constraint uses a margin to encourage more confident classification and more precise regression.
- Results: 49.4 mAP is achieved with a single ResNet-101 model, and 51.2 mAP with SENet154, on COCO and OpenImage evaluations.The method was validated with ablation studies on the standard COCO benchmark and large-scale OpenImage dataset.
2. Methods
TSD revisits the sibling head by separating classification and localization in the spatial dimension, using task-specific proposals and feature branches derived from a shared proposal. A progressive constraint further improves this disentanglement during training.
- TSD framework: TSD generates separate proposals ˆPc and ˆPr from shared proposal P for classification and localization, respectively.The proposals are estimated from the RoI feature of P and feed parallel task-specific branches.
- Task-specific branches: The disentangled branches use separate feature extractors and task heads, with the first fully connected layer shared to reduce parameters.The classification branch predicts categories and the localization branch regresses boxes from their respective task-specific feature maps.
- Task-aware spatial disentanglement learning: Classification uses a pointwise deformation on a k × k grid, while localization uses a proposal-wise translation to produce its derived proposal.The localization translation applies the same offset to every pixel, whereas classification shifts sample points within individual grid cells.
- Progressive constraint: Progressive constraint jointly optimizes TSD with the sibling head and encourages performance margins between disentangled and shared proposals.The classification constraint uses class confidence, while the localization constraint compares predicted-box IoUs and is ignored for negative proposals.
- Motivation: The sibling head shares parameters and spatial clues between classification and localization despite their different translation sensitivities.Classification requires translation-agnostic features, whereas localization requires translation-aware features.
- Scope and relation to prior methods: TSD is applicable to DCN and improves performance across different backbones by disentangling classification and localization from spatial dimension and feature extractor.The paper attributes this capability to task-aware spatial disentanglement and reports excellent performance for different backbones.
3. Experiments
Experiments evaluate TSD through ablations across disentanglement choices, training variants, proposal-learning strategies, backbones, datasets, IoU thresholds, scales, and detector models.
- Ablation studies: TSD preserves shared backbone semantics while task-specific heads improve performance; spatial disentanglement further improves AP, including AP.75, over head disentanglement.Backbone disentanglement largely degrades performance, whereas TSD without PC improves over Dhead with only a slight parameter increase.
- Ablation studies: Jointly training TSD with the sibling head is complementary, indicating that derived proposals do not conflict with the original shared proposal.Only the TSD head is retained at inference.
- Ablation studies: 1.5 AP improvement at AP.75 from progressive constraint is accompanied by barely affected AP.5, while strict AP gains 1.3 AP.The result supports PC’s focus on more confident classification and more precise regression for accurate boxes.
- Generalization: TSD improves performance by 3%–5% with approximately 10% additional time cost across backbones; ResNet-50+TSD outperforms ResNet-152 despite fewer parameters.Subsequent TSD results include TSD+PC unless otherwise specified.
- Generalization: TSD generalizes to OpenImage, Mask R-CNN, stricter IoU thresholds, and varied object scales, with increasing gains at higher IoU and especially strong results for medium and large objects.On COCO test-dev, single-model ResNet-101-DCN reaches AP 49.4 and SENet154-DCN reaches AP 51.2.
4. Conclusion
The paper presents TSD to address conflict in the sibling head by learning task-aware spatial disentanglement, with progressive constraint providing additional gains across detectors and datasets.
- TSD derives separate proposals from a shared proposal to learn task-specific representations for classification and localization.
- Progressive constraint enlarges the performance margin between disentangled and shared proposals, providing additional performance gain.
- 3%∼5%: TSD consistently boosts most backbones and models on COCO and large-scale OpenImage.
- TSD serves as the core model of the authors’ 1st solution in the OpenImage Challenge 2019.