Source-linked AI summary
Dynamic Head: Unifying Object Detection Heads with Attentions
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, Lei Zhang
TL;DR
Object detection heads lack a unified way to handle scale, spatial, and task variation. The paper proposes Dynamic Head, which applies complementary attention mechanisms across feature levels, spatial locations, and channels, achieving strong COCO results and broad detector gains.
Problem
Prior detection-head studies focus on individual scale-, spatial-, or task-awareness problems, leaving a unified solution open.
Method
Dynamic Head separately applies attention across feature levels, spatial locations, and channels to unify scale-, spatial-, and task-awareness.
Results
1.2% ∼3.2% AP gains across popular detectors and 54.0% AP with ResNeXt-101-DCN were reported on COCO.
Takeaways & Limitations
The dynamic head is presented as an efficient plugin block that can be integrated into existing object detector frameworks.
Takeaways & Limitations
The authors identify making full attention easy to learn and efficient to compute, and incorporating more attention modalities, as directions for further improvement.
Abstract
from arXiv · showhide
The complex nature of combining localization and classification in object detection has resulted in the flourished development of methods. Previous works tried to improve the performance in various object detection heads but failed to present a unified view. In this paper, we present a novel dynamic head framework to unify object detection heads with attentions. By coherently combining multiple self-attention mechanisms between feature levels for scale-awareness, among spatial locations for spatial-awareness, and within output channels for task-awareness, the proposed approach significantly improves the representation ability of object detection heads without any computational overhead. Further experiments demonstrate that the effectiveness and efficiency of the proposed dynamic head on the COCO benchmark. With a standard ResNeXt-101-DCN backbone, we largely improve the performance over popular object detectors and achieve a new state-of-the-art at 54.0 AP. Furthermore, with latest transformer backbone and extra data, we can push current best COCO result to a new record at 60.6 AP. The code will be released at https://github.com/microsoft/DynamicHead.
1. Introduction
Object detection heads must jointly address scale, spatial, and task variation, but prior studies typically focus on only one of these challenges. The dynamic head unifies the three forms of awareness through separate attention mechanisms and improves detection performance across models.
- Challenges: Object detection heads need scale-awareness, spatial-awareness, and task-awareness because images contain varied object scales, shapes, locations, and task representations.These representations include bounding boxes, centers, and corner points with different objectives and constraints.
- Research gap: Prior studies generally address only one of the three awareness problems, leaving unified treatment of all three open.
- Dynamic head: The dynamic head treats a backbone output as a level × space × channel tensor and unifies scale-, spatial-, and task-awareness with attention learning.A full self-attention solution would be difficult to optimize and computationally unaffordable.
- Attention design: Separate level-wise, spatial-wise, and channel-wise attention modules enhance feature levels, spatial representations, and task-specific channels, respectively.The modules target object scale, discriminative spatial locations, and tasks such as classification, box regression, and center/key-point learning.
- Results: 1.2% ∼3.2% AP gains were obtained across object detection models on MS-COCO, with 54.0% AP using a ResNeXt-101-DCN backbone.The method also reached 60.6 AP with a transformer backbone and extra self-training data.
2. Related Work
Related work motivates three complementary forms of awareness in detection: adaptive feature-level selection, spatially discriminative aggregation, and channel-wise task specialization. The dynamic head integrates these properties into one attention-based framework for diverse detector types.
- Scale-awareness: Scale-awareness addresses objects with different sizes, and the dynamic head adaptively adjusts the importance of feature levels to the input.
- Spatial-awareness: Spatial-aware attention applies to each spatial location and adaptively aggregates feature levels to learn more discriminative representations.
- Task-awareness: Task-aware attention operates on channels to favor different tasks across single-stage, two-stage, box-based, center-based, and keypoint-based detectors.
- Unified framework: The dynamic head integrates scale-, spatial-, and task-awareness into a unified attention mechanism for a general detection-head framework.
3. Our Approach
Dynamic Head unifies scale-, spatial-, and task-aware processing by applying sequential attention mechanisms to different dimensions of a feature tensor. The resulting blocks can be stacked and integrated into one- and two-stage detectors.
- Dynamic Head: Unifying with Attentions: The feature pyramid is resized to a common median-level scale and reshaped into F ∈ R^L×S×C, where S = H × W.L denotes pyramid levels, while H, W, and C denote spatial dimensions and channels.
- Dynamic Head: Unifying with Attentions: Dynamic Head represents a unified detection head as sequential attentions over feature levels, spatial locations, and channels.These dimensions correspond to scale-awareness, spatial-awareness, and task-awareness.
- Dynamic Head: Unifying with Attentions: A full self-attention over all tensor dimensions is computationally costly, so Dynamic Head applies three separate attentions on individual dimensions.The factorized design avoids directly learning attention over the high-dimensional tensor.
- Dynamic Head: Unifying with Attentions: Scale-aware attention dynamically fuses features at different scales according to their semantic importance.A linear function approximated by a 1 × 1 convolution and a hard-sigmoid function are used in this module.
- Dynamic Head: Unifying with Attentions: Spatial-aware attention uses deformable convolution for sparse sampling and aggregates features across levels at corresponding spatial locations.Learned spatial offsets and importance scalars focus sampling on discriminative regions.
- Dynamic Head: Unifying with Attentions: Task-aware attention dynamically switches feature channels on or off to favor different detection tasks and representations.Its hyperfunction learns activation thresholds after global average pooling, fully connected layers, normalization, and shifted-sigmoid normalization.
4. Experiment
Experiments on MS-COCO evaluate the dynamic head through ablations, efficiency tests, generalization across detectors, and comparisons with different backbones and state-of-the-art systems. The results show consistent accuracy gains, modest computational cost, and strong benchmark performance.
- Experimental Setup: MS-COCO experiments use standard AP evaluation, with ablations conducted on val2017 and training performed on train2017 without extra data.The implementation is based on Mask R-CNN benchmark code and commonly uses the ATSS framework.
- Ablation Study: 3.6 AP: the full dynamic head improves the baseline, while scale-aware, spatial-aware, and task-aware modules individually add 0.9, 2.4, and 1.3 AP.Combining scale-aware and spatial-aware modules yields a 2.9 AP improvement, indicating complementary component effects.
- Ablation Study: Scale-aware attention smooths discrepancies across feature levels by shifting higher-resolution features toward lower resolution and lower-resolution features toward higher resolution.The learned scale ratios are computed across feature levels using COCO val2017 images.
- Ablation Study: Spatial-aware attention focuses feature maps more accurately on foreground objects and discriminative spatial locations as the number of attention blocks increases from 2 to 6.Before attention, backbone feature maps are noisy and poorly focused on foreground objects.
- Ablation Study: Eight stacked dynamic head blocks provide increasing benefits before saturation, while two blocks already outperform the baseline at lower computation cost.With six blocks, the added computational cost is negligible relative to the backbone while accuracy improves substantially.
- Comparison with the State of the Art: 1.2–3.2 AP: dynamic head improves popular detectors spanning two-stage, one-stage, anchor-based, anchor-free, box-based, and point-based frameworks.The evaluated detectors include Faster R-CNN, RetinaNet, ATSS, FCOS, and RepPoints.
- Comparison with the State of the Art: 1.1 AP and 1.2 AP: dynamic head exceeds BorderDet with ResNet-101 and ResNeXt-64x4d-101 backbones, respectively.The comparison uses the same settings on the COCO benchmark.
- Comparison with the State of the Art: 54.0 AP: dynamic head reaches a new state of the art with multi-scale training and testing, exceeding concurrent best methods by 1.3 AP.With only multi-scale training, it reaches 52.3 AP using a 2x training schedule and uses 1/20 of EfficientDet and SpineNet training time.
5. Conclusion
The paper presents dynamic head as a unified framework combining scale-aware, spatial-aware, and task-aware attentions, and as a flexible plugin for object detectors. It also identifies easier full-attention learning, efficient computation, and additional attention modalities as directions for improvement.
- Dynamic head unifies scale-aware, spatial-aware, and task-aware attentions within a single object detection head framework.
- The dynamic head is designed as a flexible plugin block that can integrate into existing object detector frameworks.
- The paper highlights easier full-attention learning, efficient computation, and systematic inclusion of more attention modalities as future improvements.
- The appendix includes comparisons using transformer backbones on COCO validation and with latest methods on COCO test-dev.
Appendix
Additional experiments show that dynamic head remains effective with transformer backbones and benefits from larger inputs and self-training data. These settings improve comparison results and produce a 60.6 AP COCO record.
- Dynamic head improves a transformer-backbone baseline by 2.5 AP on downstream object detection performance.
- Dynamic head is competitive with a method requiring extra mask ground-truth when paired with transformer-based backbones.
- Larger input sizes and extra self-training data further benefit dynamic head performance.
- 60.6 AP is reported as a new COCO state-of-the-art result using larger-scale data and self-training pseudo-labels.