Source-linked AI summary
DETRs with Hybrid Matching
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, Han Hu
TL;DR
DETR’s one-to-one matching preserves end-to-end, NMS-free inference but gives few queries positive supervision. H-DETR adds one-to-many matching during training, consistently improving DETR-based methods across vision tasks while retaining the original inference branch.
Problem
One-to-one matching assigns few queries as positive samples, limiting the training efficacy of DETR-based approaches.
Method
H-DETR combines the original one-to-one matching branch with an auxiliary one-to-many branch during training, then uses the one-to-one branch for inference.
Results
H-DETR consistently improves representative DETR methods across vision tasks, including a +1.7% mAP gain for Deformable-DETR on COCO object detection.
Takeaways & Limitations
The hybrid scheme increases positive-query training efficacy while preserving one-to-one matching’s NMS-free end-to-end inference.
Takeaways & Limitations
Naive self- and cross-attention implementations increase GPU memory consumption, although optimized implementations can reduce this overhead.
Abstract
from arXiv · showhide
One-to-one set matching is a key design for DETR to establish its end-to-end capability, so that object detection does not require a hand-crafted NMS (non-maximum suppression) to remove duplicate detections. This end-to-end signature is important for the versatility of DETR, and it has been generalized to broader vision tasks. However, we note that there are few queries assigned as positive samples and the one-to-one set matching significantly reduces the training efficacy of positive samples. We propose a simple yet effective method based on a hybrid matching scheme that combines the original one-to-one matching branch with an auxiliary one-to-many matching branch during training. Our hybrid strategy has been shown to significantly improve accuracy. In inference, only the original one-to-one match branch is used, thus maintaining the end-to-end merit and the same inference efficiency of DETR. The method is named H-DETR, and it shows that a wide range of representative DETR methods can be consistently improved across a wide range of visual tasks, including DeformableDETR, PETRv2, PETR, and TransTrack, among others. The code is available at: https://github.com/HDETR
1. Introduction
DETR-based methods generalize end-to-end set matching across vision tasks, but one-to-one matching provides positive supervision to few queries. H-DETR adds one-to-many training supervision while retaining one-to-one inference and improves results across tasks.
- Motivation: One-to-one matching assigns only one query to each ground-truth object, leaving most queries with classification-only supervision.In Deformable-DETR, fewer than 30 of 300 queries typically match ground truth for each image.
- Hybrid matching: H-DETR adds an auxiliary one-to-many matching branch that assigns multiple queries to each positive sample during training.The original one-to-one branch remains available for inference.
- Hybrid matching: Inference uses only the original one-to-one decoder branch, preserving DETR’s avoidance of NMS and inference efficiency.The method introduces no additional computation overhead during evaluation.
- Results: H-DETR achieves consistent gains across representative DETR methods, tasks, and model sizes.Evaluated examples include Deformable-DETR, PETRv2, PETR, and TransTrack.
- Results: +1.7% mAP improves Deformable DETR on COCO object detection from 47.0% to 48.7%.H-DETR also improves PETR on COCO pose estimation by +1.6% mAP, from 69.3% to 70.9%.
2. Related work
Related work extends DETR architectures and label-assignment strategies across detection, segmentation, 3D perception, pose estimation, and tracking. H-DETR differs from denoising-based methods by targeting insufficient positive-query training through end-to-end one-to-many assignment.
- DETR architectures: Deformable-DETR improves DETR with multi-scale deformable attention, iterative refinement, and dynamic queries.It retains DETR’s one-to-one bipartite matching.
- DETR across vision tasks: DETR-based methods have been developed for segmentation, 3D object detection, pose estimation, tracking, and other vision tasks.These approaches adapt DETR to pixel, 3D-scene, keypoint, and spatiotemporal object localization.
- Label assignment: Label assignment methods are grouped into one-to-many and one-to-one strategies.DETR belongs to the one-to-one assignment path described in this related-work discussion.
- Auxiliary-query methods: DN-DETR and DINO-DETR introduce auxiliary queries mainly to address instability in Hungarian assignment.Their designs use noisy ground-truth augmentations and manual assignment.
- Auxiliary-query methods: H-DETR instead addresses insufficient positive-sample training with a simpler end-to-end assignment strategy using repeated ground truth and Hungarian matching.This design is presented as more general than methods based on denoising.
3. Our Approach
H-DETR augments the standard DETR pipeline with one-to-many matching while retaining one-to-one matching. It supports branch-, epoch-, and layer-based variants, combining their losses during training and using a designated query pathway for evaluation.
- Preliminary: DETR extracts pixel embeddings, decodes them with object queries, predicts after decoder layers, and performs one-to-one bipartite matching.Task-specific prediction heads generate a set of predictions independently at each decoder layer.
- Hybrid Matching: The hybrid scheme combines one-to-one matching for NMS-free prediction with one-to-many matching for more positive queries and higher training efficacy.The approach is designed to preserve the respective benefits of both assignment strategies.
- Hybrid Branch Scheme: The hybrid branch scheme maintains separate query groups for one-to-one and one-to-many matching.The two groups are processed with shared decoder layers and their losses are combined during training.
- Hybrid Branch Scheme: One-to-many matching repeats each ground truth K times before bipartite matching with predictions from the auxiliary query group.The resulting loss uses the repeated target set for auxiliary supervision.
- Hybrid Branch Scheme: The combined training objective is λL_one2many + L_one2one, while evaluation keeps only the one-to-one query branch.Masked self-attention processes both query groups in parallel to avoid their interactions.
- Hybrid Epoch Scheme: The hybrid epoch variant uses one-to-many matching for the first ρ training epochs and one-to-one matching for the remaining (1−ρ) epochs.It uses the same query group and applies the original ground truth during the one-to-one phase.
- Hybrid Layer Scheme: The hybrid layer variant applies one-to-many matching to the first L1 decoder layers and one-to-one matching to the remaining L2 layers.The decoder depth satisfies L=L1+L2.
4. Experiment
Experiments show that hybrid matching consistently improves DETR-based models across 2D and 3D detection, pose estimation, and tracking, while preserving favorable efficiency trade-offs. Ablations examine matching configuration, computational cost, localization and false negatives, and architectural choices.
- 2D object detection: H-Deformable-DETR improves Deformable-DETR from 51.8% to 53.2% with Swin-T under 12 training epochs and from 49.3% to 50.6% under 36 epochs on COCO.Gains are reported across ResNet50, Swin-T, and Swin-L backbones and multiple training schedules.
- 3D object detection: H-PETRv2 raises nuScenes validation NDS from 50.68% to 52.38%, while GPU memory increases from 7235M to 11175M.The reported improvement is associated with improved localization accuracy for multi-view 3D detection predictions.
- Multi-person pose estimation: H-PETR improves COCO pose-estimation AP from 73.3% to 74.9% with a Swin-L backbone under 100 training epochs.The paper reports consistent gains over PETR baselines.
- Multi-object tracking: H-TransTrack improves MOT17 validation MOTA from 67.1% to 68.7%, with gains related to much lower false-negative rates.Because MOT17 results show relatively large variance, the paper reports mean performance over approximately three runs.
- Efficiency: The hybrid branch scheme adds about 7% training time after merging matching and loss computations, and is selected for its faster training and inference trade-off.For H-Deformable-DETR, GFLOPs increase by 1.6%/5.4% and training time by 6.4%/23.1% for Swin-L/ResNet50, respectively.
- Ablation study: Ablations find gains arise mainly from more accurate localization and fewer false negatives; on COCO, K must exceed 3, with K=6 and T=1500 chosen by default.Independent encoder layers cause significant performance drops, whereas independent decoder layers cause only slight drops; the authors identify encoder optimization as key.
- Ablation study: Compared with one-to-many matching throughout training and NMS at evaluation, the hybrid one-to-one branch is preferred when training time and inference speed are considered.The one-to-many branch with NMS has comparable performance, while one-to-many-only matching achieves slightly better performance than the hybrid approach.
- System-level comparison: H-Deformable-DETR reaches 59.4% on COCO validation among the reported Swin-L DETR-based comparisons, including additional enhancement techniques.The system-level comparison uses single-scale evaluation and similar-capacity backbones.
5. Conclusion
The paper presents hybrid matching to address low training efficacy in DETR-based approaches while retaining one-to-one matching's NMS-free advantage and one-to-many matching's increased positive queries.
- H-DETR combines one-to-one matching, which avoids NMS, with one-to-many matching, which increases positive queries and training efficacy.
- The hybrid matching scheme is reported as simple and effective across multiple vision tasks.
- Inference retains the one-to-one matching branch, preserving the end-to-end detection design.
A. Datasets
The paper evaluates its methods on datasets spanning 2D and 3D detection, pose estimation, and multi-object tracking, using task-specific validation and test protocols.
- COCO provides 123K images for object detection and more than 200K images for pose estimation, with validation performance reported.
- nuScenes contains 1000 driving sequences collected from 6 cameras, 1 lidar, and 5 radars, split into 700, 150, and 150 sequences for train, validation, and test.
- nuScenes 3D detection is measured using nuScenes Detection Score (NDS) and mean Average Precision (mAP).
- ScanNetV2 includes 1513 indoor scenes, with 1201 used for training and 312 for validation, and reports mAP at IoU thresholds 0.25 and 0.5.
- MOT17 contains 14 pedestrian tracking videos, using 7 for training and 7 for testing, with the second half of each training video forming validation data.
B. More Hyper-parameter Details
The paper refers readers to Table 14 for detailed hyper-parameter settings used with the hybrid branch approach across DETR-based approaches and benchmarks.
- Table 14 provides detailed hyper-parameter settings for applying the hybrid branch approach.
- The settings cover different DETR-based approaches and benchmarks.
- The passage directs readers to the table rather than listing the individual settings in the text.
C. Panoptic Segmentation Results
The hybrid matching scheme improves Mask-Deformable-DETR for panoptic segmentation across model sizes and training durations, with a reported Swin-L gain of +0.3% mAP after 50 epochs.
- H-Mask-Deformable-DETR applies the hybrid branch scheme to a Mask-Deformable-DETR baseline formed by adding a mask prediction head.
- The hybrid matching scheme consistently improves ResNet50, Swin-T, and Swin-L across various training epochs.
- +0.3% mAP is reported for H-Mask-Deformable-DETR over Mask-Deformable-DETR with Swin-L after 50 epochs.
D. More Ablation Results
The ablations examine training behavior, matching design, parameter sharing, and precision-recall effects of hybrid matching. They show improved optimization and recall, while identifying localization quality and selection strategy as important constraints.
- Training behavior: Longer training lowers training loss and improves training AP, but validation loss and AP saturate from 50 to 75 epochs.This indicates that extending the baseline schedule alone does not improve validation performance.
- Matching selection: The loss-weight ablation varies λ for the one-to-many loss under K = 6 and a 12× training schedule, with λ = 1 selected by default.The authors report that the method is not sensitive to λ.
- Precision-recall analysis: Precision-recall curves show that hybrid matching mainly improves recall under two IoU thresholds, thereby lowering false-negative rates.The improvement is reported relative to the baseline.
- Matching selection: Replacing top 0–300 predictions with top 300–600 predictions drops performance from 47% to 42.5%, indicating poorer localization quality in the latter group.An improved selection scheme reaches 47.9% for K = 1 but provides no advantage at K = 6 over the original strategy.
- Training behavior: The hybrid approach decreases loss values on both training and validation sets compared with Deformable-DETR.The comparison uses the one-to-one loss throughout training for fairness.
- Parameter sharing: Independent transformer decoders and heads cause a 0.1% drop, while additionally using an independent encoder causes a 0.5% drop.The reported comparison changes from 49.1% to 49.0%, then to 48.5%.
E. Accelerate Hybrid Matching
The original hybrid implementation computes one-to-one and one-to-many matching separately, increasing per-epoch training time. Merging cost-matrix and loss computations reduces this added latency.
- Acceleration: Merging the one-to-one and one-to-many cost-matrix and loss computations accelerates hybrid matching compared with separate implementations.The separate implementation increases epoch time from 75 minutes to 85 minutes relative to the baseline.