PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Distinct Query for End-to-End Object Detection

Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, Kai Chen

arXiv:2303.12776v2cs.CV

TL;DR

Sparse queries can miss objects, while dense queries create similar predictions that complicate one-to-one optimization. DDQ combines dense initial queries with distinct-query selection, improving FCN, R-CNN, and DETR performance, including 52.1 AP for DDQ-DETR on COCO in 12 epochs.

  • Problem

    Sparse query designs do not guarantee high recall, while dense queries create similar predictions that make one-to-one optimization difficult.

  • Method

    DDQ densely distributes initial queries and selects distinct queries for one-to-one assignment across FCN, R-CNN, and DETR architectures.

  • Results

    DDQ improves multiple detector architectures; DDQ-DETR achieves 52.1 AP on COCO in 12 epochs, while DDQ detectors surpass alternatives on CrowdHuman metrics.

  • Takeaways & Limitations

    Dense and distinct queries combine advantages of traditional and end-to-end detectors across FCN, R-CNN, and DETR architectures.

  • Takeaways & Limitations

    Using many distinct queries can impose substantial memory costs, with Sparse R-CNN requiring around 45G per GPU for 7000 queries.

Abstract

from arXiv · show

One-to-one label assignment in object detection has successfully obviated the need for non-maximum suppression (NMS) as postprocessing and makes the pipeline end-to-end. However, it triggers a new dilemma as the widely used sparse queries cannot guarantee a high recall, while dense queries inevitably bring more similar queries and encounter optimization difficulties. As both sparse and dense queries are problematic, then what are the expected queries in end-to-end object detection? This paper shows that the solution should be Dense Distinct Queries (DDQ). Concretely, we first lay dense queries like traditional detectors and then select distinct ones for one-to-one assignments. DDQ blends the advantages of traditional and recent end-to-end detectors and significantly improves the performance of various detectors including FCN, R-CNN, and DETRs. Most impressively, DDQ-DETR achieves 52.1 AP on MS-COCO dataset within 12 epochs using a ResNet-50 backbone, outperforming all existing detectors in the same setting. DDQ also shares the benefit of end-to-end detectors in crowded scenes and achieves 93.8 AP on CrowdHuman. We hope DDQ can inspire researchers to consider the complementarity between traditional methods and end-to-end detectors. The source code can be found at \url{https://github.com/jshilong/DDQ}.

1. Introduction

End-to-end detection must balance recall from densely distributed queries against optimization challenges caused by sparse or redundant queries. DDQ addresses this by selecting distinct queries from dense proposals before one-to-one assignment, improving multiple detector architectures and crowded-scene performance.

  • Motivation: Traditional detectors use dense queries and one-to-many assignment for high recall, but require NMS to remove redundant predictions.This creates a mismatch between training and inference and prevents an end-to-end pipeline.
  • Motivation: DETR enables end-to-end detection with one-to-one assignment, but its limited sparse queries can fall short in recall.DETR initializes only hundreds of learnable object queries compared with densely distributed conventional queries.
  • Motivation: Dense queries under one-to-one assignment introduce similar queries that receive contradictory labels, making optimization difficult and inefficient.Such queries may represent the same instance while being assigned opposite labels.
  • Contribution: DDQ combines dense distribution for object coverage with query distinctness for one-to-one optimization across FCN, R-CNN, and DETR architectures.The method applies distinct-query selection before label assignment and extends to DDQ FCN, DDQ R-CNN, and DDQ DETR.

2. Related Work

Object detectors have primarily paired dense queries with one-to-many assignment or sparse queries with one-to-one assignment, each defining a different route toward detection.

  • Dense Queries with One-To-Many Assignment: Traditional one-stage and proposal-based detectors use dense queries with one-to-many assignments, requiring postprocessing to remove duplicate predictions.Assignment strategies range from IoU- or center-based rules to prediction-aware dynamic methods.
  • Sparse Queries with One-To-One Assignment: DETR uses a small set of learned queries with one-to-one assignment to form an end-to-end pipeline.Sparse R-CNN similarly reformulates queries as bounding boxes with corresponding embeddings.

3. Analysis of Sparse and Dense Queries

Sparse queries limit recall, whereas dense queries create similar predictions that hinder one-to-one optimization. Distinct-query selection alleviates this difficulty, although scaling query counts can become memory-intensive.

  • Query-density analysis: Sparse R-CNN performance rises as queries increase toward about 2000, indicating that its default approximately 300 queries are insufficient for recall.The experiment varies the query count from 10 to 7000.
  • Optimization difficulty: Identical queries can receive opposite foreground and background labels under one-to-one assignment, reducing or reversing useful classification gradients.The gradient is scaled down when 0 < p < 0.5 and may cause negative training when p > 0.5.
  • Distinct-query selection: Class-agnostic NMS selects distinct queries before one-to-one assignment, improving optimization and performance as the number of queries grows.The performance margin over Sparse R-CNN without selection consistently increases with more queries.
  • Practical constraint: 7000 queries require around 45G memory per GPU, motivating architecture-specific DDQ designs with more practical computation costs.The memory footprint becomes significant when using many distinct queries.

4. Method

DDQ lays dense queries, models their interactions efficiently, and selects distinct queries before one-to-one assignment and refinement. The principle is adapted to FCN, R-CNN, and DETR architectures with auxiliary training mechanisms for dense queries.

  • Paradigm of DDQ: DDQ directly uses feature-map points as dense initial queries and processes them with lightweight convolutional or linear networks in a sliding-window manner.An 800x800 input can yield more than 10000 feature-pyramid queries.
  • Distinct Queries: Class-agnostic NMS identifies distinct queries by predicted-box overlap, and loss is computed only on the selected queries during both training and inference.This differs from traditional NMS, which is used only as inference postprocessing.
  • Loss design: Auxiliary loss trains dense queries with soft one-to-many assignment, supplying gradients to filtered or otherwise unused queries.The auxiliary head follows the main head while allowing more positive samples and denser gradients.
  • DDQ FCN: DDQ FCN uses pyramid shuffle to exchange S channels between adjacent feature levels, with interpolation aligning spatial dimensions.The operation addresses missing cross-level interaction in the dense feature pyramid.
  • DDQ FCN: DDQ FCN selects top-1000 predictions per feature level before applying class-agnostic NMS at threshold 0.7.The selection is designed to preserve distinctness and generality across datasets.
  • DDQ R-CNN and DDQ DETR: DDQ R-CNN forms refinement queries from filtered DDQ FCN features, while DDQ DETR maps distinct encoder features and boxes into content and position embeddings for refinement.DDQ DETR applies distinct-query selection before each of its six refinement stages.

5. Experiments

The experiments evaluate DDQ across FCN, R-CNN, and DETR architectures on MS-COCO and CrowdHuman, using progressive comparisons and standard metrics. DDQ improves accuracy and crowded-scene performance while retaining efficient end-to-end optimization.

  • Experimental design: Experiments compare DDQ with conventional and end-to-end detectors on MS-COCO and CrowdHuman, including progressive architecture studies and benchmark comparisons.MS-COCO reports bounding-box AP; CrowdHuman reports AP, mMR, and Recall.
  • Progressive development: 41.5 AP is achieved by DDQ FCN under the standard 1x schedule, while DDQ R-CNN reaches 44.6 AP on MS-COCO.The FCN result follows the addition of an auxiliary loss for dense queries; the R-CNN result uses DDQ FCN in the early refinement stages.
  • Crowded-scene comparison: DDQ surpasses the compared detectors on all CrowdHuman metrics, combining dense queries for leading Recall with distinctness for the lowest mMR.The crowded-scene comparison concerns heavily occluded instances and evaluates AP, mMR, and Recall.
  • COCO comparison: On COCO comparisons, DDQ remains advantageous across end-to-end detectors and reaches 51.0 AP for DDQ R-CNN with a DyHead encoder.The reported 51.0 AP uses six DyHead blocks as the encoder structure.

6. Ablation study

The ablations examine recall, IoU-threshold robustness, and the sensitivity of end-to-end detection to distinct-query selection. DDQ improves query recall with lower latency and remains robust across a broad threshold range.

  • Recall improvement: DDQ R-CNN queries achieve recall comparable to Sparse R-CNN with 7000 queries, while using much less latency.Sparse R-CNN with 300 queries has 10.2 AR100.
  • DQS threshold robustness: DDQ FCN/R-CNN performance is quite robust when the DQS IoU threshold ranges from 0.6 to 0.8.Performance drops slightly below 0.6 because overlapping-object recall decreases, and degrades above 0.8 because similar queries are insufficiently suppressed.
  • Recall improvement: Table 6 reports recall improvement for dense distinct queries, with Q denoting queries, DQS distinct queries selection, and L model latency.The table caption identifies the compared quantities but does not provide numerical entries here.
  • DQS threshold robustness: DDQ uses class-agnostic NMS during training and inference to filter distinct queries, contrasting with ATSS sensitivity to its post-processing NMS IoU threshold.Distinctness continues improving DDQ DETR performance even with CDN training.

8. Conclusions

The paper concludes that both sparse and dense queries are problematic for end-to-end detection and proposes queries that are simultaneously dense and distinct. This paradigm improves FCN, R-CNN, and DETR detectors by combining advantages of traditional and end-to-end approaches.

  • 8. Conclusions: DDQ combines dense distribution for object coverage with distinctness for one-to-one label-assignment optimization.The paper presents dense distinct queries as the expected query paradigm for end-to-end object detection.
  • 8. Conclusions: DDQ significantly improves the performance of FCN, R-CNN, and DETR architectures.The conclusion frames this improvement as blending advantages from traditional detectors and recent end-to-end detectors.

A. Analysis of DDQ in Deformable DETR

The Deformable DETR analysis tests distinct-query selection under a standard 1x COCO setting and finds that filtering similar queries improves performance as query counts grow. The auxiliary loss uses selected low-cost positives with classification-weighted regression.

  • Distinct-query selection: Under the standard 1x COCO setting, naively increasing Deformable DETR queries eventually hurts performance, whereas DQS yields an increasingly larger margin.Without DQS, performance decreases around 5000 queries because training becomes more difficult with similar queries.
  • Distinct-query selection: DQS filters similar queries before each iterative-refinement stage, retaining distinct queries for Deformable DETR.The comparison is described in Figure 5.
  • Design principle: DDQ is proposed as a principle for designing object detectors with fast convergence based on recent end-to-end detectors.The principle is identified as Dense Distinct Queries.
  • Auxiliary loss: The auxiliary loss selects K lowest-cost samples per ground truth as positives and reweights each sample’s GIoU loss by its classification target.For DDQ FCN, K=8; for DDQ DETR, K=4; performance is stable for DDQ FCN when K ranges from 5 to 16.
  • Auxiliary loss: The auxiliary classification target is defined using the selected positive samples’ classification scores and IoUs.The supplied equation passage gives the target expression, while the surrounding text identifies the selected-positive set.

C. More Analysis and Ablation for Pyramid Shuffle

The pyramid-shuffle analysis compares feature-level interactions and lightweight channel configurations. It shows that retaining current-level information is important, while excessive shuffling causes a substantial performance drop.

  • Pyramid shuffle analysis: Pyramid shuffle is analyzed against 3D MAX Filtering through quantitative comparison, score-map visualization, and varying shuffle-channel settings.The analysis uses Table 9, Figure 6, and Table 11.
  • Pyramid shuffle analysis: The score maps compare pyramid shuffles and no pyramid shuffles across adjacent feature levels, with red circles marking duplicate predictions.Rainbow colors encode classification scores; blue and yellow backgrounds distinguish the two conditions.
  • Shuffle-channel ablation: The number of shuffle channels can be reduced to 16 when cross-level distinct-query selection is already present, making the design more lightweight.The supplied passage attributes this to existing cross-level selection.
  • Shuffle-channel ablation: Table 10 reports one-stage DDQ performance for different shuffle-channel counts, while Table 8 reports latency in milliseconds at batch size 1.The supplied captions identify the evaluated dimensions without listing table values.

C.1. Number of Pyramid Shuffle Operations

Pyramid shuffle operations materially affect DDQ FCN stability and performance, while DDQ R-CNN uses an encoder with dynamic blocks for stronger feature representation.

  • C.1. Number of Pyramid Shuffle Operations: 41.0 AP is the average performance without pyramid shuffle, but results fluctuate between 40.8 AP and 41.1 AP.The no-shuffle configuration is marked unstable.
  • C.1. Number of Pyramid Shuffle Operations: 0.5 AP is lost when using cross-level distinct-query selection instead of two and one pyramid shuffle operations in the two branches.The comparison concerns the classification and regression branches, respectively.
  • C.1. Number of Pyramid Shuffle Operations: Six dynamic DyHead blocks are added as the DDQ R-CNN encoder to provide a more powerful feature representation for the decoder head.The configuration adopts 500 queries and three refinement stages.
  • C.1. Number of Pyramid Shuffle Operations: DDQ components can be combined with DINO, while DINO's contrastive denoising training is not intended to address optimization among very similar dense queries.Mix query selection increases query distinctness through content embeddings, but its position embeddings can remain similar.

F. Number of Queries and Stages in DDQ RCNN

DDQ R-CNN performance depends on coordinating query count with refinement depth: more queries introduce lower-quality candidates that require additional refinement.

  • F. Number of Queries and Stages in DDQ RCNN: Table 12 evaluates DDQ R-CNN across different numbers of refinement stages and queries.
  • F. Number of Queries and Stages in DDQ RCNN: The best number of refinement stages is proportional to the number of queries in DDQ R-CNN.Increasing query count adds lower-quality queries, making more refinement stages necessary.
  • F. Number of Queries and Stages in DDQ RCNN: Two refinement stages and 300 queries are selected to trade off DDQ R-CNN performance and latency.

G. Latency Benchmark

The supplied passages describe latency measurement conditions and query-construction ablations, alongside robustness comparisons for distinct-query selection and traditional NMS in crowded scenes.

  • G. Latency Benchmark: Latency is measured as forwarding plus post-processing with batch size 1 on a Tesla A100 GPU and Intel Xeon Gold 6348 CPU.Reported latency averages 200 inference iterations using PyTorch 1.9.0, CUDA Toolkit 11.1, and cuDNN 8.0.5.
  • G. Latency Benchmark: 43.2 AP results when all DDQ R-CNN query embeddings are zero tensors and refinement stages receive only meaningful query boxes.The query-construction ablation is reported in Table 13.
  • G. Latency Benchmark: 1.0 AP is lost when DDQ R-CNN queries are constructed from FPN results, while regression-branch construction causes a 0.6 AP drop.
  • G. Latency Benchmark: DQS robustness is evaluated across different IoU thresholds in CrowdHuman, while ATSS shows a performance bottleneck despite threshold adjustment for NMS.
  • G. Latency Benchmark: Table 14 compares DDQ with different DQS IoU thresholds against ATSS post-processing thresholds, including configurations without DQS or post-processing.
  • G. Latency Benchmark: DDQ's crowded-scene performance may enable applications that violate human privacy, such as surveillance.
Loading 2303.12776v2…