Source-linked AI summary
Efficient DETR: Improving End-to-End Object Detector with Dense Prior
Zhuyu Yao, Jiangbo Ai, Boxun Li, Chi Zhang
TL;DR
DETR-style detectors require multiple decoder refinements, and the paper investigates whether random object-container initialization explains this dependence. Efficient DETR initializes containers from dense proposals and achieves competitive detection with a 3-encoder, 1-decoder design.
Problem
DETR-style detectors depend on cascaded decoder layers, while the role of randomly initialized object queries and reference points in this requirement remains to be understood.
Method
Efficient DETR combines dense prediction and sparse set detection, using top-K dense proposals and encoder features to initialize reference points and object queries before one decoder layer.
Results
Efficient DETR achieves competitive performance with 3 encoder layers and 1 decoder layer, including 44.2AP on COCO with ResNet50 and 36 training epochs.
Takeaways & Limitations
The findings support using dense prior to initialize object containers, narrowing the performance gap between one-decoder and multi-decoder end-to-end detection.
Takeaways & Limitations
The paper's stated structural setting includes a 6-encoder and 6-decoder transformer architecture for DETR and Deformable DETR.
Abstract
from arXiv · showhide
The recently proposed end-to-end transformer detectors, such as DETR and Deformable DETR, have a cascade structure of stacking 6 decoder layers to update object queries iteratively, without which their performance degrades seriously. In this paper, we investigate that the random initialization of object containers, which include object queries and reference points, is mainly responsible for the requirement of multiple iterations. Based on our findings, we propose Efficient DETR, a simple and efficient pipeline for end-to-end object detection. By taking advantage of both dense detection and sparse set detection, Efficient DETR leverages dense prior to initialize the object containers and brings the gap of the 1-decoder structure and 6-decoder structure. Experiments conducted on MS COCO show that our method, with only 3 encoder layers and 1 decoder layer, achieves competitive performance with state-of-the-art object detection methods. Efficient DETR is also robust in crowded scenes. It outperforms modern detectors on CrowdHuman dataset by a large margin.
1. Introduction
The paper identifies randomly initialized object containers as a key reason DETR-style detectors require multiple decoder refinements, then proposes Efficient DETR, which uses dense predictions to initialize them.
- Modern detectors use dense anchors and NMS, whereas end-to-end detection aims to predict bounding boxes without these hand-crafted components.
- DETR and Deformable DETR use cascaded decoder layers to iteratively refine randomly initialized object containers.Object containers include object queries and reference points.
- Reference points represent guessed box centers and behave like anchor points, while their initialization strongly affects one-decoder performance.
- Efficient DETR combines dense and sparse detection, using top-K dense proposals and encoder features to initialize reference points and object queries.The method uses K=100 proposals in this paper.
- With a 3-encoder and 1-decoder Deformable DETR structure, Efficient DETR achieves 44.2AP on COCO with ResNet50 and 36 training epochs.
2. Related Work
Related detectors differ in how they generate and refine object candidates: mainstream methods use dense anchors or proposals, while end-to-end methods predict sets without extra post-processing.
- One-stage detectors densely predict categories and offsets for anchors across feature maps, commonly using one-to-many assignment and NMS.
- Two-stage detectors first generate class-agnostic foreground proposals with an RPN, then extract proposal features using ROIPool or ROIAlign.
- End-to-end detectors use encoder-decoder transformers and object queries to iteratively interact with encoder features without extra post-processing stages.
- Deformable DETR uses multi-scale features and local sampling around reference points to improve small-object detection and convergence.
- Table 1 compares encoder and decoder configurations in a Res50 Deformable DETR with 100 proposals and a 3× training schedule.
- Table 2 evaluates how changing the number of decoder layers affects DETR performance.
3. Exploring DETR
DETR’s decoder cascade is highly sensitive to initialization: reference points and object queries provide the object-container state that iterative refinement improves. Dense region proposals provide a more suitable initial state, substantially improving the 1-decoder structure.
- Encoder and decoder: Removing two decoder layers causes about a 9.3 AP drop, whereas removing two encoder layers causes only a 1.7 AP drop.The comparison uses a DETR with 3 encoders and 3 decoders as baseline.
- Decoder cascade: A 1-layer decoder trails a 6-layer decoder by 10.3 AP, consistent with randomly initialized object queries requiring iterative refinement.Object queries are updated after each iteration, and final predictions are produced from them by the detection head.
- Reference points: Reference points are 2-d tensors representing guessed box centers and, after six iterations, gather from image-wide coverage toward foreground centers.They act like anchor points and focus deformable attention on critical sampling points around foreground objects.
- Reference-point initialization: Different reference-point initializations produce large performance differences with one decoder but converge to similar final distributions after six iterations.Grid initialization approximately matches learnable initialization, while center and border initialization suffer large non-cascade accuracy drops.
- Dense prior: RPN region proposals initialize reference points from proposal centers, yielding a more reasonable distribution and a large improvement in the non-cascade structure.The RPN predicts objectness scores and anchor offsets; top-scoring boxes become region proposals.
- Object-query initialization: Initializing object queries with corresponding 256-d encoder features from dense proposals further improves the 1-decoder structure by 3 AP.Together, proposal-based reference points and query features form the dense-prior initialization used by Efficient DETR.
4. Efficient DETR
Efficient DETR combines dense proposal generation with sparse transformer refinement, using dense prior to initialize reference points and object queries before a single decoder layer.
- Architecture: Efficient DETR uses 3 encoder layers and 1 decoder layer, replacing the decoder cascade with dense-to-sparse object-container initialization.The dense and sparse parts share a detection head, and the sparse decoder further refines initialized containers.
- Dense prior initialization: Top-K selection chooses scored 4-d proposals and paired 256-d encoder features as reference points and object queries.The selected proposal features and reference points remain paired through their shared anchor.
- Sparse refinement: The sparse decoder uses initialized object containers to interact with encoder features and produce refined final predictions.Cross-attention aggregates features relevant to each initialized query before prediction.
- Proposal scheduling: The training process starts with many proposals and dynamically reduces their number to improve foreground coverage and efficiency.The method begins with 300 proposals and can achieve comparable accuracy with only 100 proposals.
5. Experiments
Experiments on COCO, ablations, and CrowdHuman evaluate Efficient DETR’s accuracy, efficiency, proposal sensitivity, decoder depth, and crowded-scene robustness.
- Main results: 44.2 AP with 36 epochs on COCO: Efficient DETR outperforms Faster RCNN and most end-to-end detectors with fewer FLOPs and parameters.It is 0.4 AP above Deformable DETR and only 0.3 AP below Sparse RCNN, while using 32M rather than 40M parameters.
- Main results: Efficient DETR converges in 36 epochs versus 500 for DETR, while using a 3-encoder and 1-decoder structure instead of 6 encoders and 6 decoders.This preserves fast convergence while simplifying the transformer architecture.
- Ablation study: Class-specific dense prediction improves accuracy by providing category information in encoder features used to initialize sparse object queries.Sharing the detection head between dense and sparse parts does not harm performance.
- Ablation study: Increasing proposals from 100 to 1000 yields only a 0.2 AP improvement, while linearly decreasing proposals during training improves stability.One-to-one assignment supports high accuracy with a small proposal set, whereas one-to-many assignment requires more proposals.
- Ablation study: Adding three encoder layers improves performance by 0.9 AP, whereas stacking decoder layers brings slight gains and can degrade performance after two iterations.These results indicate that decoder cascading is unnecessary for Efficient DETR.
- Crowded-scene evaluation: On CrowdHuman, Efficient DETR gains 4 AP and 5 mMR over Deformable DETR using only 100 proposals.Using 400 proposals increases mMR, consistent with additional false positives in crowded scenes.
6. Conclusion
The paper attributes DETR’s need for repeated refinement to randomly initialized object containers and proposes dense-prior initialization as a simpler alternative.
- Conclusion: Reference points serve as anchor points for object queries, clarifying the role of object containers in DETR-style detectors.Object containers include abstract object features and positional information.
- Conclusion: Efficient DETR combines dense detection and set detection to achieve high performance with fast convergence.The conclusion presents the pipeline as a simple and efficient end-to-end detector.