PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank-DETR for High Quality Object Detection

Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, Gao Huang

arXiv:2310.08854v3cs.CVcs.LG

TL;DR

Ranking mechanisms remain an underexplored aspect of DETR-based detection, despite their importance to AP at high IoU thresholds. Rank-DETR combines rank-oriented architecture and optimization designs. On COCO val, it reaches 50.2% AP in 12 epochs, surpassing H-DETR by +1.5% and DINO-DETR by +1.2% AP.

  • Problem

    Few studies specifically investigate accurate ranking for DETR-based detectors, although ranking matters for average precision at high IoU thresholds.

    Source: §1 · p. 1

  • Method

    Rank-DETR combines rank-aware classification and query architecture with a loss and matching cost designed to prioritize better-localized predictions.

    Source: §1 · p. 2

  • Results

    Rank-DETR achieves 50.2% AP on COCO val with 12 training epochs, surpassing H-DETR [30] by +1.5% and DINO-DETR [75] by +1.2% AP.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The paper concludes that rank-oriented architecture and optimization designs improve DETR performance under high IoU thresholds and are complementary.

    Source: §3.4 · p. 6, §5 · p. 10

Abstract

from arXiv · show

Modern detection transformers (DETRs) use a set of object queries to predict a list of bounding boxes, sort them by their classification confidence scores, and select the top-ranked predictions as the final detection results for the given input image. A highly performant object detector requires accurate ranking for the bounding box predictions. For DETR-based detectors, the top-ranked bounding boxes suffer from less accurate localization quality due to the misalignment between classification scores and localization accuracy, thus impeding the construction of high-quality detectors. In this work, we introduce a simple and highly performant DETR-based object detector by proposing a series of rank-oriented designs, combinedly called Rank-DETR. Our key contributions include: (i) a rank-oriented architecture design that can prompt positive predictions and suppress the negative ones to ensure lower false positive rates, as well as (ii) a rank-oriented loss function and matching cost design that prioritizes predictions of more accurate localization accuracy during ranking to boost the AP under high IoU thresholds. We apply our method to improve the recent SOTA methods (e.g., H-DETR and DINO-DETR) and report strong COCO object detection results when using different backbones such as ResNet-$50$, Swin-T, and Swin-L, demonstrating the effectiveness of our approach. Code is available at \url{https://github.com/LeapLabTHU/Rank-DETR}.

1 Introduction

DETR-based detection has advanced, but ranking mechanisms remain underexplored despite their importance to average precision at high IoU thresholds. Rank-DETR combines rank-oriented architecture and optimization designs, with reported gains across strong DETR-based methods.

1 Introduction

  • DETR-based methods have also been extended to segmentation, pose estimation, and multi-object tracking.The paper situates these extensions within continued work to improve DETR-based object detection.Source: §1 · p. 1
  • Ranking mechanisms for DETR-based detectors have received little focused study, although accurate ranking matters especially for average precision at high IoU thresholds.Prior DETR work has focused on encoder-decoder architectures and query formulations.Source: §1 · p. 1
  • Rank-DETR combines rank-adaptive classification and query-rank layers with rank-oriented loss and matching cost designs.The classification head adjusts scores with rank-aware learnable logit biases, while the query rank layer adds ranking embeddings; the optimization designs prioritize boxes with more accurate localization.Source: §1 · p. 2
  • +2.1% AP75 (52.9% vs. 55.0%) with ResNet-50 and +2.7% AP75 (55.1% vs. 57.8%) with Swin-T are reported over H-DETR.Rank-DETR also reaches 50.2% AP in the 1× schedule on COCO val.Source: §1 · p. 2

2 Related Work

Prior DETR work has developed architectural and query-formulation extensions, while object-detection ranking research has explored score and loss designs. Rank-DETR focuses on rank-oriented DETR architecture and optimization, including a query-wise dynamic approach.

DETR for Object Detection

  • Deformable-DETR, DAB-DETR, DN-DETR, and DINO-DETR advanced DETR through attention, query formulation, and one-to-one matching changes.DETR also removes the need for hand-designed components such as non-maximum suppression.Source: §2 · p. 2

Ranking for Object Detection

  • Object-detection ranking work includes IoU prediction, IoU-aware classification, quality focal loss, anchor alignment, and ranking-based losses.Stable-DINO and Align-DETR also apply IoU-aware classification ideas to DINO-DETR loss and matching design.Source: §2 · p. 2

Dynamic Neural Networks

  • Rank-DETR introduces a query-wise dynamic approach that integrates ranking information into object queries based on box-quality ranking.Its rank-oriented architecture complements rank-oriented matching-cost and loss designs.Source: §2 · p. 2

3 Approach

Rank-DETR adds rank-aware query and classification components to DETR, then aligns classification scores and matching with localization quality. The design targets more accurate ranking, particularly at high IoU thresholds.

3.1 Preliminary

  • DETR encodes image features and object queries through a Transformer decoder, predicts classes and boxes at each layer, then matches predictions one-to-one with ground truth.The detectors described rank dense encoder predictions to initialize queries and return the highest-confidence decoder predictions at evaluation.Source: §3.1 · p. 3

3.2 Rank-oriented Architecture Design: ensure lower FP and FN

  • Rank-adaptive classification heads follow every decoder layer, while query rank layers precede the last L−1 layers to adjust scores and regenerate queries.The rank-aware content query combines sorted object content queries with learned static content embeddings; positional queries are sorted or recreated to match that order.Source: §3.2 · pp. 3–4
  • For H-DETR, positional queries are sorted; for DINO-DETR, ranked box predictions are used to recreate positional-query embeddings.The regenerated content and positional queries feed the following decoder layer.Source: §3.2 · p. 4
  • oLRPFP fell 24.5%→24.1% and oLRPFN 39.5%→38.6% with rank-aware logit biases and content-query vectors.These learnable representations adjust classification scores using query ranking information.Source: §3.2 · p. 5

3.3 Rank-oriented Matching Cost and Loss: boost the AP under high IoU thresholds

  • GIoU-aware classification uses normalized GIoU targets, while the high-order matching cost combines classification confidence and IoU to prioritize localization accuracy.The matching cost increases the influence of localization accuracy with a larger α; it is introduced from the middle training stage because early predictions are poorly localized.Source: §3.3 · p. 5
  • AP75 rose 52.9%→54.1% with GIoU-aware classification loss and 52.9%→54.0% with high-order matching cost.The high-order cost is used from the middle training stage, when predictions have better localization quality.Source: §3.3 · pp. 5–6

3.4 Discussion

  • Unlike earlier GIoU-aware classification losses [31] [37] [76] and concurrent rank-oriented cost and loss work [1] [45], Rank-DETR adds a rank-oriented architecture and reports complementarity between the designs.The discussion identifies rank-adaptive classification and query ranking as the architectural distinction.Source: §3.4 · p. 6

4 Experiment

On COCO, Rank-DETR improves results over H-DETR and DINO-DETR, including at higher IoU thresholds. Ablations examine its components, showing gains from rank-oriented designs, GIoU-aware classification, and high-order matching.

4.1 Experiment Setting

  • Experiments train on COCO train and evaluate on val, following the original H-DETR and DINO-DETR setups.Source: §4.1 · p. 6

4.2 Main Results

  • 50.2% AP on COCO val with 12 epochs surpasses H-DETR by +1.5% and DINO-DETR by +1.2%.Source: §4.2 · p. 6, Table 1 · p. 6
  • With H-DETR, Rank-DETR improves AP across backbones and schedules, with AP75 gains of +2.1%, +2.7%, and +1.9% for ResNet-50, Swin-Tiny, and Swin-Large.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Ablation Study and Analysis

  • Progressively adding components improves performance, and RCH raises AP by +0.2% on H-DETR and +0.4% in the complete method, with larger gains in AP75.Source: §4.3 · p. 7, Table 4 · p. 8
  • QRL improves performance by +0.3% on H-DETR and +0.7% in the complete method, while reducing oLRPFP from 24.5% to 23.8%.Source: §4.3 · pp. 7–8, Table 4 · p. 8
  • GCL gains 0.7% mAP over the vanilla baseline; normalized GIoU targets outperform IoU targets by better modeling non-overlapping boxes.Source: §4.3 · pp. 8–9, Table 5 · p. 8
  • HMC improves performance by +0.6% and +0.4% in two ablations; exponent-4 IoU cost peaks, and HMC lowers unmatched-query overlap with ground truth.Source: Table 4 · p. 8, §4.3 · p. 9
  • Rank-DETR improves detection with only slight FLOPs and inference-latency increases, while positive-query scores rise and hard-negative scores fall across decoder layers.Source: §4.3 · p. 10

5 Conclusion

Rank-DETR applies rank-oriented designs to the architecture and optimization of DETR detectors. The paper reports improved performance under high IoU thresholds and identifies further exploration of rank-oriented designs as an open direction.

5 Conclusion

  • Rank-DETR combines rank-oriented designs to improve modern object detectors within the DETR framework.The conclusion describes the approach as incorporating ranking information into both network architecture and optimization.Source: §5 · p. 10
  • More precise prediction ranking is the paper’s core insight for robust performance under high IoU thresholds.Source: §5 · p. 10
  • The paper identifies further exploration of rank-oriented designs as an open direction for building high-quality object detectors.Source: §5 · p. 10

Related papers

Loading 2310.08854v3…