PaperlayerView exact revision on arXiv

Source-linked AI summary

DEIM: DETR with Improved Matching for Fast Convergence

Shihua Huang, Zhichao Lu, Xiaodong Cun, Yongjun Yu, Xiao Zhou, Xi Shen

arXiv:2412.04234v3cs.CVcs.AI

TL;DR

DETR's one-to-one matching yields sparse supervision, while randomly initialized queries create low-quality matches that slow convergence. DEIM combines Dense O2O matching with Matchability-Aware Loss to increase positive samples and better optimize matches across quality levels. On COCO, it improves real-time DETR performance while halving training costs, including 53.2% mAP in one day on an NVIDIA 4090 GPU.

  • Problem

    DETR's O2O matching provides only one positive sample per target, and randomly initialized queries produce numerous low-quality matches, limiting effective supervision.

  • Method

    DEIM combines Dense O2O matching, which increases targets and positive samples through augmentation, with MAL, which emphasizes low-quality matches using IoU and confidence.

  • Results

    DEIM improves RT-DETRv2 and D-FINE performance while halving training costs, achieving 53.2% mAP with a ResNet50-based DETR in approximately one day on one NVIDIA 4090 GPU.

  • Takeaways & Limitations

    DEIM provides a training framework for real-time DETR models that improves accuracy and training efficiency without requiring additional data.

  • Takeaways & Limitations

    DEIM-trained models still show a slight small-object AP disadvantage compared with YOLO models.

Abstract

from arXiv · show

We introduce DEIM, an innovative and efficient training framework designed to accelerate convergence in real-time object detection with Transformer-based architectures (DETR). To mitigate the sparse supervision inherent in one-to-one (O2O) matching in DETR models, DEIM employs a Dense O2O matching strategy. This approach increases the number of positive samples per image by incorporating additional targets, using standard data augmentation techniques. While Dense O2O matching speeds up convergence, it also introduces numerous low-quality matches that could affect performance. To address this, we propose the Matchability-Aware Loss (MAL), a novel loss function that optimizes matches across various quality levels, enhancing the effectiveness of Dense O2O. Extensive experiments on the COCO dataset validate the efficacy of DEIM. When integrated with RT-DETR and D-FINE, it consistently boosts performance while reducing training time by 50%. Notably, paired with RT-DETRv2, DEIM achieves 53.2% AP in a single day of training on an NVIDIA 4090 GPU. Additionally, DEIM-trained real-time models outperform leading real-time object detectors, with DEIM-D-FINE-L and DEIM-D-FINE-X achieving 54.7% and 56.5% AP at 124 and 78 FPS on an NVIDIA T4 GPU, respectively, without the need for additional data. We believe DEIM sets a new baseline for advancements in real-time object detection. Our code and pre-trained models are available at https://github.com/ShihuaHuang95/DEIM.

1. Introduction

DEIM addresses DETR's slow convergence by increasing positive samples through Dense O2O matching and improving low-quality matches with MAL. Experiments show faster training and higher performance for real-time DETR models.

  • Motivation: DETR's O2O matching provides sparse supervision because each target receives only one positive sample, slowing convergence.This is especially limiting for small objects, where dense supervision is important.
  • DEIM: Dense O2O increases positive samples by adding targets through standard augmentations while retaining one-to-one matching.Mosaic and mixup can generate additional targets per image without additional decoders.
  • DEIM: MAL scales penalties using IoU and classification confidence, emphasizing low-quality matches that existing losses under-optimize.It behaves similarly to VFL for high-quality matches while assigning greater weight to low-quality ones.
  • Results: 53.2% mAP is achieved by a ResNet50-based DETR on one NVIDIA 4090 GPU in approximately one day.With half the training epochs, DEIM exceeds RT-DETRv2 by 0.2 AP and D-FINE by 0.6 AP.
  • Results: 50% lower training cost accompanies improved performance when DEIM is applied to existing real-time DETRs.The framework is described as simple and flexible, combining Dense O2O with MAL.

2. Related Work

Related work improves DETR supervision, match quality, and computational efficiency through modified assignments, query initialization, losses, and efficient encoders. DEIM instead combines denser target construction with a loss designed for low-quality matches.

  • Increasing positive samples: Group DETR, H-DETR, and Co-DETR increase supervision by assigning multiple queries or auxiliary one-to-many labels, but may add computational overhead.Group DETR uses independent O2O matching across query groups, while Co-DETR uses auxiliary heads based on detector assignments.
  • Optimizing low-quality matches: Prior-informed query initialization improves query distributions around objects but can leave nonsalient objects with low-quality matches.Low-quality matches remain a significant challenge even with advanced initialization methods.
  • Optimizing low-quality matches: VFL reduces uncertainty between classification confidence and box quality but under-optimizes low-IoU matches because their loss values remain small and flat.DEIM introduces MAL to better optimize these matches.
  • Reducing computation cost: Efficient attention and encoder designs reduce DETR computation, including deformable attention, hybrid CNN-attention encoders, and iterative regression.RT-DETR and D-FINE build real-time detection systems around these efficiency improvements.

3. Method

DEIM increases supervision in DETR by expanding one-to-one matching across more targets, then uses Matchability-Aware Loss to handle the resulting low-quality matches. The framework retains O2O structure while improving sensitivity to match quality.

  • O2M vs. O2O: O2M assigns multiple queries to each target, whereas O2O uses one Hungarian-matched prediction per target and therefore provides sparser supervision.O2O is a special case of O2M with M_i = 1 for every target.
  • Loss functions: Focal loss uses class labels and confidence but does not account for bounding-box localization quality.Its parameters γ and α control easy-versus-hard sample emphasis and foreground-versus-background weighting, respectively.
  • O2M vs. O2O: O2O produces fewer positive matches than O2M, with O2M sometimes exceeding 80 matches per image and reaching about 10 times as many matches in extreme cases.The comparison uses Hungarian matching for O2O and SimOTA for O2M on RT-DETRv2 with ResNet50 on MS COCO.
  • Dense O2O: Dense O2O retains one-to-one matching while increasing the number of targets through image composition, producing denser supervision without changing the matching structure.One example replicates an image into four quadrants and combines them into one composite image, increasing targets from 1 to 4.
  • Loss functions: VFL incorporates IoU into query-quality optimization but gives low-quality matches little loss and treats zero-overlap matches as negatives, limiting their training value.These limitations are more pronounced in DETR because its queries are sparse and matching is rigid.
  • Loss functions: MAL incorporates matching quality into the loss, modifies the target label from q to q^γ, and removes α to reduce overemphasis on high-quality boxes.For IoU = 0.05, MAL increases loss more sharply with confidence than VFL; for IoU = 0.95, the two losses perform similarly.

4. Experiments

Experiments evaluate DEIM across COCO and CrowdHuman, real-time detector comparisons, augmentation strategies, and component ablations. DEIM improves convergence and detection performance while reducing training costs, though small-object performance remains slightly below YOLO models.

  • Comparisons with real-time detectors: DEIM-D-FINE-L achieves 54.7 vs. 54.1 AP and reduces inference time by 20% versus YOLOv11-X.The reported inference times are 8.07 ms for DEIM-D-FINE-L and 10.74 ms for YOLOv11-X.
  • Comparisons with ResNet-based DETRs: DEIM reduces RT-DETRv2 training time by half while increasing AP by 0.5 with ResNet-50 and 0.9 with ResNet-101.It surpasses all compared variants after 36 epochs and exceeds DINO-Deformable-DETR by 2.7 AP with ResNet-50.
  • Comparisons with ResNet-based DETRs: DEIM improves small-object AP by 1.3 with ResNet-50 and 2.1 with ResNet-101 compared with the corresponding RT-DETRv2 models.The passage reports comparable overall AP while highlighting stronger gains for small objects.
  • Comparisons on CrowdHuman: 1.5 AP improvement over D-FINE-L is achieved on CrowdHuman, with gains greater than 3% on small objects and AP75.Both D-FINE-L and its DEIM-enhanced version are trained for 120 epochs.
  • Analysis: Mosaic and mixup increase positive samples and improve performance after 12 epochs, while combining them accelerates convergence further.Dense O2O therefore strengthens supervision by increasing target counts per image using computationally efficient augmentations.
  • Analysis: Dense O2O reaches baseline-like performance after 36 rather than 72 epochs, and adding MAL further improves performance through better box quality.The ablation evaluates RT-DETRv2-R50 and D-FINE-L.

5. Conclusion

DEIM accelerates DETR-based real-time detection by improving both the quantity and quality of matches. The resulting framework improves accuracy and training speed without compromising inference latency.

  • 5. Conclusion: DEIM combines Dense O2O matching with MAL to increase positive samples and optimize matches across varying quality levels.Dense O2O increases target supervision, while MAL specifically enhances low-quality matches.
  • 5. Conclusion: DEIM achieves superior performance in fewer epochs than models such as YOLOv11 and shows gains over RT-DETR and D-FINE.The conclusion frames these gains as improvements in both detection accuracy and training speed.
  • 5. Conclusion: DEIM improves training efficiency and detection accuracy without compromising inference latency for real-time applications.The conclusion identifies these properties as the method’s practical advantages.

1. Experimental Settings

Experiments use COCO with standard detection metrics and evaluate DEIM within D-FINE and RT-DETRv2 frameworks. Training uses augmentation and learning-rate scheduling choices tailored to Dense O2O.

  • 1. Experimental Settings: COCO train2017 is used for training and val2017 for validation, with AP, AP50, AP75, APS, APM, and APL reported.AP averages IoU thresholds from 0.50 to 0.95 in 0.05 steps.
  • 1. Experimental Settings: DEIM is implemented and validated using the D-FINE and RT-DETRv2 frameworks, with most hyperparameters following their original settings.Framework-specific differences are provided in Tables 9 and 10.
  • 1. Experimental Settings: The proposed data augmentation scheduler is illustrated alongside the FlatCosine learning-rate scheduler for 60 training epochs.The scheduler is tailored for Dense O2O training.

2. Comparison with Lighter YOLO Detectors

DEIM improves lightweight real-time detectors across RT-DETRv2 and D-FINE model sizes, with approximately 1 AP gains in RT-DETRv2 and a 1.3 AP gain for the M⋆ variant.

  • Approximately 1 AP improvement occurs across all three RT-DETRv2 model sizes with DEIM.
  • 1.3 AP is the gain achieved by DEIM-RT-DETRv2-M⋆.
  • DEIM achieves state-of-the-art results compared with other lightweight real-time detection methods.

3. Additional Results

Additional experiments examine implementation changes, supervision density, training behavior, and qualitative performance. Dense O2O increases positive samples, while an intermediate target count improves performance more than either lower or higher counts.

  • After 36 epochs, minor modifications leave D-FINE-L unchanged but improve D-FINE-X by 0.1 AP, from 55.4 to 55.5.The modified configuration becomes the new experimental baseline.
  • Dense O2O significantly increases the number of positive samples during one training epoch.The comparison uses the same training images with and without Dense O2O.
  • Performance improves from 10 objects per training image to 25 but drops at 50.The decline is attributed to positive-to-negative imbalance and data-distribution shift caused by too many objects.
  • DEIM achieves higher validation accuracy and slightly lower training accuracy, indicating reduced training-set overfitting and better adaptability to new samples.

4. Visualizations

Qualitative comparisons show that DEIM addresses duplicated high-confidence predictions and false positives observed in D-FINE-L.

  • DEIM addresses D-FINE-L’s high-confidence duplicated predictions and false positives in qualitative comparisons.Examples include removing four overlapping kite boxes and correcting object misclassifications while detecting the bottle.
Loading 2412.04234v3…