Source-linked AI summary
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment
Qiang Chen, Xiaokang Chen, Jian Wang, Shan Zhang, Kun Yao, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng, Jingdong Wang
TL;DR
DETR’s one-to-one assignment supports end-to-end detection, but naive one-to-many assignment does not work for DETR training. Group DETR introduces grouped one-to-many assignment with separate self-attention, and reports faster convergence and improved performance across DETR-based models while retaining standard inference.
Problem
Applying one-to-many supervision to DETR remains challenging because naive one-to-many assignment does not work for DETR training.
Method
Group DETR uses multiple object-query groups, performs one-to-one assignment within each group, and applies separate self-attention to each group.
Results
5.0 mAP improvement is reported for Conditional DETR-C5 with 12-epoch COCO training, alongside consistent improvements across various DETR-based methods.
Takeaways & Limitations
Group DETR accelerates DETR training convergence while keeping inference unchanged and requiring only one query group.
Takeaways & Limitations
Naive one-to-many assignment performs poorly because it lacks a mechanism to promote one prediction and demote duplicate predictions for each object.
Abstract
from arXiv · showhide
Detection transformer (DETR) relies on one-to-one assignment, assigning one ground-truth object to one prediction, for end-to-end detection without NMS post-processing. It is known that one-to-many assignment, assigning one ground-truth object to multiple predictions, succeeds in detection methods such as Faster R-CNN and FCOS. While the naive one-to-many assignment does not work for DETR, and it remains challenging to apply one-to-many assignment for DETR training. In this paper, we introduce Group DETR, a simple yet efficient DETR training approach that introduces a group-wise way for one-to-many assignment. This approach involves using multiple groups of object queries, conducting one-to-one assignment within each group, and performing decoder self-attention separately. It resembles data augmentation with automatically-learned object query augmentation. It is also equivalent to simultaneously training parameter-sharing networks of the same architecture, introducing more supervision and thus improving DETR training. The inference process is the same as DETR trained normally and only needs one group of queries without any architecture modification. Group DETR is versatile and is applicable to various DETR variants. The experiments show that Group DETR significantly speeds up the training convergence and improves the performance of various DETR-based models. Code will be available at \url{https://github.com/Atten4Vis/GroupDETR}.
1. Introduction
DETR enables end-to-end detection through one-to-one assignment, but applying one-to-many supervision remains difficult. Group DETR addresses this with grouped queries and separate self-attention, accelerating convergence while preserving standard DETR inference.
- DETR uses one-to-one assignment to promote one prediction per ground-truth object and demote duplicates during training.
- Naive one-to-many assignment succeeds in traditional detectors but does not work for DETR training.
- Group DETR assigns objects one-to-one within each of K query groups, creating one-to-many supervision while keeping predictions competitive within groups.Separate self-attention prevents predictions in different groups from influencing one another.
- Inference remains the same as normally trained DETR and requires only one query group, without architecture modification.
- The grouped architecture is equivalent to parallel parameter-sharing decoders and resembles learned object-query augmentation that introduces more supervision.The approach is also reported to improve encoder training, presumably through the improved decoder.
2. Background
DETR combines an encoder, decoder, and class-and-box predictors, using query interactions and bipartite one-to-one matching to produce set-based predictions without NMS. Conventional one-to-many detectors instead assign each object to multiple anchors or pixels and remove duplicates with NMS.
- DETR consists of an encoder, transformer decoder, and object class and box position predictors.The encoder maps image I to image features X, which the decoder processes with object queries.
- The decoder maps image features and object queries to query embeddings and then predictions.Its layers use self-attention, cross-attention, and feed-forward processing.
- DETR performs bipartite matching between unordered predictions and ground-truth objects, producing a one-to-one correspondence.The loss sums classification and box-regression losses over the matched prediction–ground-truth pairs.
- Decoder self-attention lets predictions exchange information needed to promote one prediction and demote duplicates for each object.This query interaction works jointly with one-to-one assignment for end-to-end detection without NMS.
- Traditional non-end-to-end detectors assign one ground-truth object to multiple anchors or pixels and use NMS during inference to remove duplicates.
3. Group DETR
Group DETR enables one-to-many supervision by partitioning object queries into groups, applying one-to-one assignment within each group, and separating self-attention. Shared parallel decoders provide additional supervision while preserving standard single-group inference.
- Motivation: Naive one-to-many assignment fails because duplicate predictions lack a mechanism to promote one score and suppress the others.Group DETR addresses this limitation through competition within each query group.
- Group-wise assignment: Group DETR forms K query groups and performs one-to-one bipartite matching independently within each group, assigning each ground-truth object to multiple predictions overall.Each group contains N queries and produces its own prediction set.
- Group-wise assignment: Self-attention is performed separately for each query group, while cross-attention and feed-forward operations remain parallelizable.The resulting training architecture uses K parallel decoders with shared decoder and predictor parameters.
- Inference and loss: Inference keeps the standard DETR process and uses only one query group, requiring no architecture modification.The training loss aggregates one loss from each decoder group.
- Analysis: Group DETR produces more stable assignments and improves training beyond simply adding training time, with an additional 1.5 mAP gain when combined with DN-DETR.The method is reported as complementary to DN-DETR, whose noised queries provide further gains.
- Interpretation: Group DETR can be viewed as query augmentation or simultaneous training of K parameter-sharing DETR models, thereby introducing more supervision.The augmented queries are spatially close for the same objects, and shared parameters receive gradients from the groups.
4. Experiments
Experiments show that Group DETR consistently improves DETR-based baselines, accelerates convergence, and generalizes to detection, 3D detection, and instance segmentation.
- Object Detection: Group DETR consistently improves dense- and deformable-attention DETR baselines under 12-epoch training.It improves Conditional DETR by 5.0 mAP, DAB-DETR by 3.9 mAP, DN-DETR by 2.0 mAP, and DAB-Deformable-DETR by 1.5 mAP.
- Object Detection: 58.4 mAP with Swin-Large remains 0.4 mAP above the DINO baseline after 50-epoch training.
- Object Detection: Group DETR speeds training convergence across various DETR variants.
- Object Detection: 64.5 mAP on COCO test-dev with ViT-Huge is reported for Group DETR applied to DINO.The result follows DINO’s encoder pretraining, Object365 pretraining, and COCO fine-tuning pipeline.
- Ablation Study: Group-wise assignment and separate self-attention improve Conditional DETR, while naive one-to-many assignment performs poorly.Group-wise assignment raises mAP from 32.6 to 34.8, and separate self-attention adds a further 2.8 mAP; performance stabilizes at 11 groups.
5. Related Works
Related work accelerates DETR through cross-attention modification or stabilization of one-to-one assignment, while this paper investigates one-to-many assignment for end-to-end DETR.
- Acceleration Strategies: DETR acceleration methods mainly modify cross-attention or stabilize one-to-one assignment.
- One-to-Many Assignment: One-to-many assignment is widely used in detectors such as Faster R-CNN and FCOS, but its direct application to DETR remains challenging.
- Concurrent Work: H-DETR also uses one-to-many assignment but combines one-to-one and one-to-many assignments in different query groups.Group DETR instead uses group-wise one-to-many assignment with separate self-attention and equal query counts across groups.
- Concurrent Work: DETA directly uses one-to-many assignment and restores NMS, whereas Group DETR preserves end-to-end detection without NMS.
6. Conclusion
Group DETR combines group-wise one-to-many assignment with parallel self-attention to add supervision while preserving standard DETR inference.
- Conclusion: Group-wise assignment confines competition among predictions to each query group, while separate self-attention supports training.
- Conclusion: Inference uses one query group and does not require NMS or architectural modification.
- Conclusion: The approach is presented as simple, easily implemented, and general across DETR-style techniques and vision problems.
A.1. Datasets and Evaluation Metrics
The experiments evaluate object detection and instance segmentation on COCO and multi-view 3D detection on nuScenes using standard task metrics.
- Datasets: COCO provides about 118K training images, 5K validation images, and 20K test images.
- Evaluation Metrics: COCO evaluation reports box mAP for detection and mask mAP for instance segmentation on val2017 under IoU thresholds from 0.5 to 0.95.
- Datasets and Metrics: nuScenes contains 1000 driving sequences, split into 700 training, 150 validation, and 150 test sequences.Multi-view 3D detection is evaluated on the validation set using NDS and mAP.
A.2. Implementation Details
Group DETR uses multiple groups of object queries that share the same architecture and query count. The design resembles query data augmentation and parameter-sharing network training.
- Group DETR adopts multiple groups of object queries, with each group sharing the same architecture and number of queries.
- The multi-group mechanism resembles data augmentation with automatically learned object-query augmentation.
- For one-stage DETR frameworks, implementation uses multiple groups of learnable object queries.
- Two-stage DETR frameworks require constructing multiple query-group pairs because query initialization depends on first-stage top-N predicted boxes.
A.3. More Results of DN-DETR
The additional DN-DETR results show that increasing denoising queries does not reliably improve performance, while Group DETR also benefits SAM-DETR variants. The reported SAM-DETR gains remain positive across both short and long schedules.
- Increasing the number of denoising queries in DN-DETR does not achieve further improvements and produces unstable performances.
- The DN-DETR experiments use 100 denoising queries in the main-paper experiments.
- Figure 10 reports DN-DETR detection performance as mAP on MS COCO for different numbers of denoising queries.
- Group DETR improves SAM-DETR by 3.1 mAP with 12e and 1.9 mAP with 50e.
- Group DETR improves SAM-DETR++ by 2.2 mAP with 12e and 1.3 mAP with 50e.
B. More Comparisons on COCO test-dev
The ViT-Huge system applies Group DETR within the DINO training pipeline and reaches 64.5 mAP on COCO test-dev. The reported result is below methods using larger foundation models and more pre-training data, but above several methods with larger models or more data.
- The system follows a pipeline of ImageNet-1K pre-training, Object365 detector pre-training, and COCO fine-tuning.
- 64.5 mAP is achieved on COCO test-dev with a ViT-Huge-based Group DETR system.
- The model outperforms methods with larger models and more pre-training data, while EVA and InterImage-H report 64.7 mAP and 65.4 mAP.
- Query-feature fusion across scales gives an approximately 0.8 mAP improvement.
- Table 9 evaluates Group DETR on SAM-DETR and SAM-DETR++ using ResNet-50 on COCO val2017.