PaperlayerView exact revision on arXiv

Source-linked AI summary

DETRs with Collaborative Hybrid Assignments Training

Zhuofan Zong, Guanglu Song, Yu Liu

arXiv:2211.12860v6cs.CV

TL;DR

DETR’s one-to-one matching provides too few positive queries, producing sparse encoder supervision and inefficient decoder attention learning. Co-DETR adds one-to-many-supervised auxiliary heads and customized positive queries, improving DETR variants across benchmarks, including 59.5% AP for DINO-Deformable-DETR with Swin-L on COCO val and 66.0% AP on COCO test-dev with ViT-L.

  • Problem

    One-to-one set matching assigns too few positive queries, limiting encoder feature discrimination and decoder attention learning.

  • Method

    Co-DETR trains parallel auxiliary heads with one-to-many assignments and supplies their positive coordinates as customized decoder queries.

  • Results

    Co-DETR improves multiple DETR variants and reaches 59.5% AP on COCO val with DINO-Deformable-DETR and Swin-L, plus 66.0% AP on COCO test-dev with ViT-L.

  • Takeaways & Limitations

    The training scheme enhances encoder and decoder training while discarding auxiliary heads during inference, adding no inference parameters or computational cost.

Abstract

from arXiv · show

In this paper, we provide the observation that too few queries assigned as positive samples in DETR with one-to-one set matching leads to sparse supervision on the encoder's output which considerably hurt the discriminative feature learning of the encoder and vice visa for attention learning in the decoder. To alleviate this, we present a novel collaborative hybrid assignments training scheme, namely $\mathcal{C}$o-DETR, to learn more efficient and effective DETR-based detectors from versatile label assignment manners. This new training scheme can easily enhance the encoder's learning ability in end-to-end detectors by training the multiple parallel auxiliary heads supervised by one-to-many label assignments such as ATSS and Faster RCNN. In addition, we conduct extra customized positive queries by extracting the positive coordinates from these auxiliary heads to improve the training efficiency of positive samples in the decoder. In inference, these auxiliary heads are discarded and thus our method introduces no additional parameters and computational cost to the original detector while requiring no hand-crafted non-maximum suppression (NMS). We conduct extensive experiments to evaluate the effectiveness of the proposed approach on DETR variants, including DAB-DETR, Deformable-DETR, and DINO-Deformable-DETR. The state-of-the-art DINO-Deformable-DETR with Swin-L can be improved from 58.5% to 59.5% AP on COCO val. Surprisingly, incorporated with ViT-L backbone, we achieve 66.0% AP on COCO test-dev and 67.9% AP on LVIS val, outperforming previous methods by clear margins with much fewer model sizes. Codes are available at \url{https://github.com/Sense-X/Co-DETR}.

1. Introduction

DETR’s one-to-one matching reduces positive-query supervision, limiting encoder feature discrimination and decoder attention learning. Co-DETR addresses this with collaborative one-to-many auxiliary assignments and customized positive queries, improving convergence and detection performance.

  • Motivation: One-to-one matching assigns each ground-truth box to one query, creating sparse supervision that harms encoder and decoder learning.The paper analyzes both latent encoder representations and decoder attention learning as consequences of too few positive queries.
  • Co-DETR: Co-DETR integrates parallel auxiliary heads supervised by one-to-many assignments such as ATSS, FCOS, and Faster RCNN.These assignments enrich supervision on the encoder output and support more discriminative latent features.
  • Co-DETR: Customized positive queries extracted from auxiliary-head coordinates provide multiple positive query–ground-truth pairs for decoder training.Positive coordinates from anchors and proposals are organized into independent groups sent to the original decoder.
  • Results: Co-DETR improves training convergence and performance across DAB-DETR, Deformable-DETR, and DINO-Deformable-DETR variants.The approach is presented as plug-and-play across these DETR architectures.
  • Results: 66.0% AP on COCO test-dev and 67.9% AP on LVIS val are achieved with a ViT-L backbone.The paper reports these as state-of-the-art results with much fewer model sizes.

2. Related Works

One-to-many assignment methods assign multiple detector outputs to each ground-truth box, using anchors, proposals, or spatial priors. Co-DETR uses such assignments through auxiliary heads to improve encoder representations while retaining an end-to-end detector framework.

  • One-to-many label assignment: One-to-many assignment assigns multiple box candidates to the same ground-truth box as positive training samples.Faster RCNN and RetinaNet use IoU-based selection, FCOS uses center priors, and ATSS uses adaptive top-k anchor selection.
  • One-to-many label assignment: Traditional one-to-many detectors include Faster RCNN, RetinaNet, FCOS, ATSS, and PAA, with different sample-selection mechanisms.These mechanisms range from fixed IoU thresholds and center priors to adaptive statistical or probabilistic selection.
  • Co-DETR: Co-DETR applies one-to-many assignments through auxiliary heads to improve encoder representations in a DETR-based detector.The method combines dense assignment supervision with the one-to-one set-matching paradigm.

3. Method

Co-DETR combines auxiliary one-to-many assignment heads with customized positive queries to enrich encoder supervision and improve decoder attention learning during training.

  • Collaborative Hybrid Assignments Training: Co-DETR introduces auxiliary heads supervised by one-to-many assignments, including Faster-RCNN, ATSS, RetinaNet, and FCOS, on transformer encoder outputs.These heads use the corresponding assignment paradigms and standard implementations such as anchor generation.
  • Collaborative Hybrid Assignments Training: Different one-to-many assignments provide denser supervision that encourages encoder features to become discriminative enough for auxiliary-head convergence.The method targets sparse encoder supervision caused by too few positive queries under one-to-one matching.
  • Customized Positive Queries Generation: Co-DETR generates extra customized positive queries from positive spatial coordinates selected by each auxiliary head.For auxiliary head i, the positive coordinates form B_i^{pos} with M_i samples, while the generated queries have dimension M_i × C.
  • Customized Positive Queries Generation: The training design creates K + 1 query groups: one original one-to-one branch and K auxiliary one-to-many branches sharing the original decoder-layer parameters.All auxiliary-branch queries are treated as positive, so matching is discarded there; the final objective balances branch losses with λ1 and λ2.
  • Why Co-DETR works: One-to-many supervision improves encoder discriminability: ATSS and Co-Deformable-DETR distinguish key-object areas better than Deformable-DETR, which is more affected by background.The comparison uses foreground and background indicators derived from discriminability scores, with resizing omitted.
  • Why Co-DETR works: Co-DETR improves matching stability and attention-related IoF-IoB curves relative to Deformable-DETR and Group-DETR, while explicitly assigning multiple spatial positives per ground truth.Unlike Group-DETR, H-DETR, and SQR, it applies dense supervision directly to latent feature maps and uses off-the-shelf one-to-many matching.

4. Experiments

Experiments show that Co-DETR consistently improves DETR variants across training schedules, backbones, and detection benchmarks, while auxiliary-head diversity must be limited to avoid optimization conflicts.

  • Main Results: Co-DETR improves Deformable-DETR from 37.1% to 42.9% AP and retains a +3.2% AP gain with 36-epoch training.Conditional-DETR and DAB-DETR also gain 2.4% and 2.3% AP, respectively, under a long training schedule.
  • Main Results: 59.5% AP is achieved by DINO-Deformable-DETR with Swin-L, improving its baseline from 58.5% AP.Co-DETR also reaches 56.9% AP with Deformable-DETR++ and exceeds that baseline by +1.7% AP.
  • Large-Scale Benchmarks: 66.0% AP on COCO test-dev is obtained with a 304M-parameter ViT-L model, surpassing InternImage-G by +0.5% AP.On LVIS, the Objects365-pretrained model reaches 67.9% AP on val and 71.9% AP on minival, with one-tenth the model size of InternImage-G.
  • Large-Scale Benchmarks: 67.9% AP is achieved on LVIS val, while the same model obtains 71.9% AP on minival and reduces model size to one-tenth of InternImage-G.The comparison uses an Objects365-pretrained Co-DETR model without elaborate test-time augmentation.
  • Ablation Studies: Performance improves as the number of auxiliary heads increases below K=3, but degradation occurs at K=6 because severe conflicts emerge among heads.The authors commonly use ATSS and Faster-RCNN with K ≤2 and avoid many diverse heads.
  • Ablation Studies: Auxiliary heads and customized positive queries both accelerate convergence and improve performance by strengthening encoder features and decoder attention learning.A complementary Faster-RCNN head reaches 49.5% AP, compared with 48.7% AP for one ATSS head and 49.2% AP for two ATSS heads.

5. Conclusions

Co-DETR is a collaborative hybrid assignment training scheme that combines versatile one-to-many supervision with DETR-based detectors. It enhances encoder and decoder training efficiency, achieving strong results on COCO and LVIS with a ViT-L backbone.

  • Co-DETR trains multiple parallel auxiliary heads with versatile one-to-many label assignments to enhance encoder learning.
  • Co-DETR extracts positive coordinates from auxiliary heads to create customized positive queries for decoder training.
  • 66.0% AP on COCO test-dev and 67.9% AP on LVIS val are achieved with a ViT-L backbone.
  • The ViT-L results establish a new state-of-the-art detector with much fewer model sizes.

Supplementary Material

The supplementary material includes figures describing relations and distances among detector heads, alongside tables documenting auxiliary-head convolution counts and loss-weight tuning.

  • Table 12 reports the influence of the number of convolutions in the auxiliary head.
  • Table 13 reports results from tuning the loss coefficients λ1 and λ2.
  • Figure 8 presents a relation matrix for the DETR, ATSS, and Faster-RCNN heads in Co-Deformable-DETR with K = 2 and ResNet-50.

A. More ablation studies

The ablation studies examine auxiliary-head architecture, loss weighting, head distances, customized positive queries, and optimization conflicts. They report robust loss coefficients, improved performance over comparison methods, and head-distance behavior as K changes.

  • One shared convolution makes the auxiliary head lightweight while retaining higher performance in 12-epoch training.
  • The loss coefficients {λ1, λ2} are robust, with the default setting {1.0, 2.0}.
  • Faster-RCNN and ATSS introduce 18.7 and 8.8 positive samples per ground-truth box, respectively, when K = 1.
  • 46.2% AP for Co-DETR with K = 1 surpasses Group-DETR’s 44.6% AP without customized positive-query generation.
  • Larger S_i,j values indicate lower consistency between heads and greater optimization inconsistency.
  • The experiments use COCO val as D and Grad-CAM as C, with CAM maps computed from DETR encoder output features.

B. More implementation details

The implementation details describe one-stage auxiliary heads based on conventional detector designs and specify how positive coordinates are assigned for anchor-free detectors.

  • One-stage auxiliary heads are evaluated using designs from ATSS, RetinaNet, FCOS, and PAA, with GIoU loss applied to the heads.
  • Reducing the stacked convolutions from four to one improves training efficiency without an accuracy drop.
  • For anchor-free FCOS heads, positive coordinates use widths and heights of 8 × 2^j.
Loading 2211.12860v6…