Source-linked AI summary
Conditional DETR for Fast Training Convergence
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang
TL;DR
DETR’s slow convergence is linked to its reliance on content embeddings for localizing object regions. Conditional DETR adds decoder-derived conditional spatial queries to narrow cross-attention, achieving 6.7× faster convergence on R50/R101 and 10× faster on DC5-R50/DC5-R101, while retaining global dense attention.
Problem
DETR suffers from slow convergence and relies heavily on content embeddings for localizing object extremities and predicting boxes.
Method
Conditional DETR learns a conditional spatial query from each decoder embedding for decoder multi-head cross-attention.
Results
6.7× faster convergence is reported for R50 and R101, and 10× faster convergence for DC5-R50 and DC5-R101.
Takeaways & Limitations
Conditional spatial queries let attention heads focus on bands containing object extremities or interior regions, relaxing dependence on content embeddings and reducing training difficulty.
Takeaways & Limitations
The approach focuses on accelerating DETR training and does not address the high computational complexity of the encoder.
Abstract
from arXiv · showhide
The recently-developed DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings for localizing the four extremities and predicting the box, which increases the need for high-quality content embeddings and thus the training difficulty. Our approach, named conditional DETR, learns a conditional spatial query from the decoder embedding for decoder multi-head cross-attention. The benefit is that through the conditional spatial query, each cross-attention head is able to attend to a band containing a distinct region, e.g., one object extremity or a region inside the object box. This narrows down the spatial range for localizing the distinct regions for object classification and box regression, thus relaxing the dependence on the content embeddings and easing the training. Empirical results show that conditional DETR converges 6.7x faster for the backbones R50 and R101 and 10x faster for stronger backbones DC5-R50 and DC5-R101. Code is available at https://github.com/Atten4Vis/ConditionalDETR.
1. Introduction
DETR achieves good object-detection performance but converges slowly because cross-attention depends heavily on content embeddings for spatial localization. Conditional DETR introduces conditional spatial queries to narrow attention and substantially accelerate training.
- DETR achieves good object-detection performance while removing many hand-crafted components, including non-maximum suppression and anchor generation.
- 500 training epochs are needed for DETR to obtain good performance, highlighting its slow training convergence.
- DETR’s cross-attention relies heavily on content embeddings, increasing the need for high-quality embeddings and training difficulty.
- Conditional DETR learns a conditional spatial embedding from each previous decoder output to form a spatial query for decoder cross-attention.
- 6.7× faster convergence is achieved with R50 and R101, while DC5-R50 and DC5-R101 converge 10× faster.
- Each cross-attention head attends to a band containing an object extremity or an interior object region, shrinking the localization range for class and box prediction.
2. Related Work
Related work addresses DETR’s computational complexity and slow convergence through sparse attention, alternative detection heads, or spatial modulation. Conditional DETR instead retains global dense attention while learning spatial representations from decoder embeddings.
- DETR removes hand-designed components such as non-maximum suppression and initial guess generation, but global encoder self-attention creates high computational complexity.
- Deformable DETR replaces decoder cross-attention with deformable attention that attends to sparse positions learned from content embeddings.
- TSP eliminates cross-attention modules and combines FCOS- and R-CNN-like detection heads to address slow convergence.
- SMCA modulates global cross-attention with Gaussian maps around learned centers, whereas conditional DETR learns conditional spatial queries from decoder content embeddings.
- Conditional DETR’s linear projection is learned from decoder embeddings to represent displacement and scaling information, unlike conditional convolution methods that learn convolutional kernels from inputs.
3. Conditional DETR
Conditional DETR modifies DETR’s decoder cross-attention by combining content information with a conditional spatial query derived from decoder embeddings and reference points. This helps heads localize distinct object regions, improving attention localization and accelerating training.
- Pipeline: Conditional DETR uses a CNN backbone, transformer encoder, transformer decoder, and class and box predictors to detect all objects end-to-end.The pipeline avoids non-maximum suppression and anchor generation.
- Decoder: Each decoder layer combines self-attention, cross-attention, and feed-forward processing to refine embeddings used for class and box prediction.Cross-attention aggregates encoder embeddings, while self-attention removes duplicate predictions.
- Conditional cross-attention: Conditional cross-attention concatenates content and spatial queries, with corresponding content and spatial keys, to separate spatial localization from content matching.The spatial query is learned from decoder information and a reference point, while content queries and keys retain appearance information.
- Conditional spatial query: The conditional spatial query maps decoder displacement information and a reference-point positional embedding into the same space as the spatial keys.The projection uses an FFN followed by element-wise scaling with a 256-dimensional parameter vector.
- Visualization and analysis: Each spatial-attention head localizes a distinct region, while combining spatial and content maps filters scattered highlights and preserves extremity regions for box regression.The learned maps also translate highlights toward object extremities or interior regions and scale their spread with object size.
4. Experiments
Experiments evaluate conditional DETR on COCO using standard DETR training and evaluation protocols, comparing backbones, training durations, related variants, and ablations. Conditional DETR reaches comparable or better performance with substantially fewer epochs, while its gains are evaluated against design alternatives and scope limitations.
- Experimental setting: Experiments use COCO 2017 with about 118K training images and 5K validation images, standard COCO average-precision evaluation, and four DETR backbones.The backbones are ResNet-50, ResNet-101, DC5-ResNet-50, and DC5-ResNet-101.
- Comparison to DETR: Conditional DETR with 75/108 training epochs outperforms DETR trained for 500 epochs across four backbones.With 50 epochs, R50 and R101 are slightly below 500-epoch DETR, while DC5-R50 and DC5-R101 perform similarly.
- Comparison to DETR: 10× faster convergence is reported for DC5-R50 and DC5-R101, versus 6.67× faster for R50 and R101.The comparison is against the original DETR trained for 500 epochs.
- Comparison to related variants: Conditional DETR scores 40.9 versus 39.4 and 43.8 versus 41.5 against deformable DETR-SS on R50 and DC5-R50, respectively.The authors caution that the comparison may be unfair because parameter and computation complexities differ.
- Comparison to multi-scale and higher-resolution variants: The approach is not designed to match multi-scale and higher-resolution DETR variants because it retains a single-scale encoder without an 8×-resolution or strong multi-scale design.Despite this scope, its DC5-R50 and DC5-R101 AP scores are reported as close to two such variants.
- Ablations: Ablations find that the proposed conditional spatial query performs best among tested query-construction schemes, while diagonal, full, and block projections perform on par.Reference points learned as parameters or predicted from object queries also outperform fixed zero reference points.
5. Conclusion
Conditional DETR uses conditional cross-attention to narrow spatial localization and reduce reliance on content queries, thereby reducing training difficulty.
- Conditional DETR learns spatial queries from reference points and decoder embeddings for decoder multi-head cross-attention.
- The learned spatial queries highlight bands around object extremities and small regions inside object boxes.
- Narrower spatial attention ranges help content queries localize distinct regions while relaxing their dependence and reducing training difficulty.
- The paper identifies human pose estimation and line segment detection as future application areas for conditional cross-attention.