PaperlayerView exact revision on arXiv

Source-linked AI summary

Object Detection with Transformers: A Review

Tahira Shehzadi, Khurram Azeem Hashmi, Didier Stricker, Muhammad Zeshan Afzal

arXiv:2306.04670v3cs.CV

TL;DR

Prior surveys largely predate recent detection-transformer advances, while DETR itself has faced slow convergence and weaker small-object detection. This review synthesizes architectural developments and compares their performance and complexity; on COCO minival, modified models improve performance and convergence, with DINO reaching 49.0% mAP at 12 epochs.

  • Problem

    Most earlier object-detection surveys predate DETR improvements, leaving a detailed review of transformer-based object detectors missing.

    Source: §2 · p. 3

  • Method

    The paper reviews DETR’s architectural advances and evaluates detection transformers on MS COCO, comparing performance, complexity, and design.

    Source: §8 · p. 20

  • Results

    On COCO minival, modified DETRs improve performance and convergence; DINO reaches mAP 49.0% at 12 epochs and performs well on small objects.

    Source: §6 · p. 17

  • Takeaways & Limitations

    The review organizes recent improvements around backbone modification, query design, and attention refinement, alongside comparisons of model advantages and limitations.

    Source: §8 · p. 20

    DETR typically uses a fixed number of object queries during training and inference despite images varying in object count.

    Source: §7 · p. 20

Abstract

from arXiv · show

The astounding performance of transformers in natural language processing (NLP) has motivated researchers to explore their applications in computer vision tasks. DEtection TRansformer (DETR) introduces transformers to object detection tasks by reframing detection as a set prediction problem. Consequently, eliminating the need for proposal generation and post-processing steps. Initially, despite competitive performance, DETR suffered from slow training convergence and ineffective detection of smaller objects. However, numerous improvements are proposed to address these issues, leading to substantial improvements in DETR and enabling it to exhibit state-of-the-art performance. To our knowledge, this is the first paper to provide a comprehensive review of 21 recently proposed advancements in the original DETR model. We dive into both the foundational modules of DETR and its recent enhancements, such as modifications to the backbone structure, query design strategies, and refinements to attention mechanisms. Moreover, we conduct a comparative analysis across various detection transformers, evaluating their performance and network architectures. We hope that this study will ignite further interest among researchers in addressing the existing challenges and exploring the application of transformers in the object detection domain. Readers interested in the ongoing developments in detection transformers can refer to our website at: https://github.com/mindgarage-shan/trans_object_detection_survey

1 INTRODUCTION

DETR reframes object detection as end-to-end set prediction with a transformer, removing proposal generation and post-processing steps. This review organizes recent DETR advances around convergence and small-object challenges, and compares their architectures and performance.

1 INTRODUCTION

  • DETR replaces region-proposal methods with an end-to-end transformer encoder-decoder and removes hand-crafted proposal and non-maximum-suppression components, [8].The review identifies slow convergence and reduced performance on small objects as limitations addressed by later modifications.Source: §1 · p. 1
  • Recent methods target DETR’s limitations through changes to attention, pretraining, queries, backbones, and feature processing [9]–[27].Examples include Deformable-DETR’s attention changes for convergence, Anchor-DETR’s anchor-point queries, and DINO’s noised queries for convergence and small-object performance.Source: §1 · pp. 1–2

1.1 Our Contributions

  • The review categorizes DETR improvements by backbone, pretraining, attention, and query design, and evaluates them on MS COCO [30].It also compares accuracy and computational complexity and discusses advantages, limitations, and future directions.Source: §1.1 · pp. 2–3
  • Table 1 summarizes DETR modifications aimed at faster training convergence and better performance on small objects.Its categories include backbone, pretraining, attention, and query modifications.Source: Table 1 · p. 3

2 RELATED PREVIOUS REVIEWS AND SURVEYS

Earlier object-detection surveys cover broad methods, specific applications, or related vision-transformer topics, but most predate improvements to DETR. The paper presents a review of ongoing detection-transformer progress as a needed resource for researchers.

2 RELATED PREVIOUS REVIEWS AND SURVEYS

  • Most prior surveys appeared before improvements to detection transformers, leaving a detailed review of transformer-based object detectors missing.Existing surveys include broad object-detection reviews, task-specific surveys, and work introducing vision transformers.Source: §2 · p. 3

3 OBJECT DETECTION AND TRANSFORMERS IN VISION

The section surveys object detection and transformer applications across vision tasks, from detection and segmentation to generation, low-level vision, and multimodal processing.

3.1 Object Detection

  • Object detection localizes and classifies image objects, commonly using bounding boxes; single-stage detectors are faster but lower-performing than two-stage networks.Backbones extract feature maps and are often pretrained on ImageNet before fine-tuning.Source: §3.1 · pp. 3–4

3.2 Transformer for Segmentation

  • Transformer encoder-decoder modules provide another approach to image segmentation, alongside self-attention methods used for pixel-level prediction.Panoptic segmentation jointly predicts per-pixel class and instance labels.Source: §3.2 · p. 4

3.3 Transformers for Scene and Image Generation

  • Transformers are used in image and scene generation, including text-conditioned image generation and transformer-based GANs.TransGAN places transformer modules in both its generator and discriminator.Source: §3.3 · p. 4

3.4 Transformers for Low-level Vision

  • Attention-based approaches have been introduced for low-level vision problems.The section relates this direction to attention networks’ performance in high-level vision.Source: §3.4 · p. 4

3.5 Transformers for Multi-Modal Tasks

  • Vision-language transformer designs include single-stream and dual-stream networks, distinguished primarily by their choice of loss functions.Applications include visual question answering, visual commonsense reasoning, cross-modal retrieval, and image captioning.Source: §3.5 · p. 4

4 DETECTION TRANSFORMERS

The section describes DETR’s encoder-decoder detection pipeline and reviews Deformable-DETR and UP-DETR modifications. These approaches alter attention and pretraining, respectively.

4.1 DETR

  • DETR uses a backbone, encoder, and decoder to predict a fixed number of objects in parallel, without anchor generation or NMS.For COCO, the number of predictions is set to 100; bipartite matching supports duplicate-free predictions.Source: §4.1 · pp. 5–6
  • Hungarian matching pairs predicted and ground-truth boxes by region and label, and SGD reduces the resulting loss.The bounding-box loss combines generalized IoU and L1 loss.Source: §4.1 · p. 5

4.2 Deformable-DETR

  • Deformable-DETR takes ten times fewer training epochs than simple DETR and uses multiscale deformable attention with a feature pyramid.The pyramid combines features at 1/8, 1/16, and 1/32 input-image resolutions.Source: §4.2 · p. 6

4.3 UP-DETR

  • UP-DETR pretrains DETR using random-sized image patches as decoder object queries and patch-feature reconstruction to protect classification features.The reconstruction uses normalized CNN-backbone patch features.Source: §4.3 · p. 6
  • UP-DETR addresses multi-query localization with an attention mask and query shuffle, dividing object queries into groups assigned to patches.Each patch is provided to N/X object queries.Source: §4.3 · p. 6

4.4 Efficient-DETR

These DETR variants target slow convergence through changes to query initialization, attention, and feature processing. WB-DETR also tests a backbone-free design that adds local patch information to transformer tokens.

4.4 Efficient-DETR

  • Efficient-DETR initializes sparse-module queries and reference points from top-k proposals generated by a dense module.The dense and sparse modules share a detection head and use one-to-one assignment.Source: §4.4 · p. 7

4.5 SMCA-DETR

  • SMCA-DETR uses query-predicted object centers and scales to weight decoder attention toward likely box regions, narrowing the search and speeding convergence.Its Gaussian-like spatial map gives more weight to locations nearer the predicted center.Source: §4.5 · pp. 7–8

4.6 TSP-DETR

  • TSP-DETR pairs an FPN-based encoder with TSP-FCOS or TSP-RCNN modules, using bipartite matching to accelerate DETR training convergence.TSP-FCOS selects top-scored features; TSP-RCNN refines regions proposed by an RPN before encoding them.Source: §4.6 · pp. 8–9

4.7 Conditional-DETR

  • Conditional-DETR forms spatial queries from 2D coordinates and prior decoder embeddings, narrowing cross-attention to explicit regions for localization.Reference points are learned for each box or generated from the corresponding object query.Source: §4.7 · p. 9

4.8 WB-DETR

  • WB-DETR removes the CNN backbone and uses LIE-T2T to reorganize adjacent image patches and model local channel relationships.The transformer processes image tokens directly, while LIE-T2T supplies local information that the transformer lacks.Source: §4.8 · p. 9

4.9 PnP-DETR

PnP-DETR reduces redundant spatial computation by sampling fine foreground features and pooling background context. Dynamic-DETR changes encoder and decoder attention to speed convergence and focus on small objects.

4.9 PnP-DETR

  • PnP-DETR separates fine foreground features from contextual background features, using poll sampling and weighted pooling to reduce transformer computation.The pool sampler aggregates remaining background features into a smaller set of context vectors.Source: §4.9 · pp. 9–10

4.10 Dynamic-DETR

  • Dynamic-DETR combines convolutional multi-scale encoder attention with an ROI-based dynamic decoder to speed convergence and improve small-object detection.The decoder uses pooled image features and box encodings with dynamic filters for cross-attention.Source: §4.10 · pp. 10–11

4.11 YOLOS-DETR

YOLOS-DETR adapts a vision-transformer sequence for detection, while Anchor-DETR and Sparse-DETR change query design and encoder-token selection. These variants reduce reliance on uniform, unfiltered image features.

4.11 YOLOS-DETR

  • YOLOS-DETR adapts ViT by replacing its classification token with 100 learnable detection tokens and using bipartite matching loss for detection.Image patches and detection tokens, with positional embeddings, form the transformer encoder sequence.Source: §4.11 · p. 11

4.12 Anchor-DETR

  • Anchor-DETR uses learned or grid anchor points as object queries and row-column decoupled attention to reduce memory cost with similar or better efficiency.Its anchor-point queries can support multiple object predictions at one region.Source: §4.12 · pp. 11–12

4.13 Sparse-DETR

  • Sparse-DETR scores encoder tokens, selects salient tokens and top-k decoder queries, and focuses on foreground tokens to reduce computation.Its scoring network selects the top ρ% of tokens before the encoder, and auxiliary heads select top-k tokens for decoder queries.Source: §4.13 · p. 12

4.14 D2ETR

These methods modify feature processing, pretraining, attention, or query design to address detection-transformer limitations. Their approaches include encoder-free architectures, refined box queries, and denoising training.

4.14 D2ETR

  • D2ETR uses fine-fused backbone features and cross-scale attention without an encoder, reporting better performance with low computational cost.Its PVT backbone combines cross-scale and intra-scale interactions before feeding fused features to the decoder.Source: §4.14 · p. 13

4.15 FP-DETR

  • FP-DETR pretrains the encoder rather than the decoder, using a multi-scale tokenizer instead of a CNN backbone, and reports competitive detection performance.The approach fully pretrains Deformable-DETR on ImageNet before detection fine-tuning; decoder pretraining is described as challenging for ImageNet classification.Source: §4.15 · pp. 13–14

4.16 CF-DETR

  • CF-DETR combines a Transformer Enhanced FPN with coarse and fine decoder blocks to refine predicted locations using global and local information.The fine block uses Adaptive Scale-Fusion and Local Cross-Attention to refine coarse boxes.Source: §4.16 · p. 14

4.17 DAB-DETR

  • DAB-DETR uses bounding-box coordinates as decoder object queries and updates them at each layer, providing positional information and a spatial prior for attention.The paper reports faster training convergence from this query design.Source: §4.17 · p. 14

4.18 DN-DETR

  • DN-DETR adds noisy ground-truth box-label queries alongside learnable matching anchors, with an attention mask separating the two query parts.The denoising part reconstructs ground-truth objects; the paper reports improved performance and training convergence.Source: §4.18 · p. 14

4.19 AdaMixer

  • AdaMixer removes the encoder and samples query-dependent features from a 3D feature space built from multi-scale CNN backbone features.Its decoder uses content and positional vectors as object queries and applies the AdaMixer mechanism to produce predictions.Source: §4.19 · pp. 14–15

4.20 REGO-DETR

REGO-DETR refines attention using RoI-based glimpses, while DINO combines mixed query selection, box prediction, and contrastive denoising. The paper reports faster convergence and improved performance for these approaches.

4.20 REGO-DETR

  • REGO-DETR uses recurrent processing of prior detections and glimpse-based RoI features to refine attention, with the paper reporting faster convergence and better performance.The method extracts glimpse features from previously detected boxes and transforms them using multi-head cross-attention.Source: §4.20 · pp. 15–16

4.21 DINO

  • DINO combines mixed object-query selection, a look-forward-twice box-prediction mechanism, and Contrastive DeNoising with an additional denoising loss.Its positive and negative anchor queries are assigned using noise thresholds; the paper reports faster convergence and improved performance for small objects.Source: §4.21 · p. 16

5 DATASETS AND EVALUATION METRICS

The evaluation compares detection-transformer modifications on the COCO2014 mini validation set using mean Average Precision. The figures describe comparisons by training epochs, model size, and object size.

5 DATASETS AND EVALUATION METRICS

  • Figure 10 compares detection transformers by training epochs and model size, using ResNet-50 results for the epoch comparison.The caption notes that DC5-labeled networks use a dilated feature map.Source: Figure 10 · p. 17
  • Figure 11 compares detection-transformer performance on small, medium, and large objects using a ResNet-50 backbone.Source: Figure 11 · p. 17
  • The evaluation uses the COCO2014 mini validation set and mean Average Precision (mAP) to compare detection-transformer improvements.The passage defines mAP as the mean of category Average Precision values, with AP measured as area under the precision-recall curve.Source: §5 · pp. 16–17

6 RESULTS AND DISCUSSION

The review compares DETR variants on performance, convergence, and model size, highlighting gains on overall and small-object detection alongside architectural trade-offs.

6 RESULTS AND DISCUSSION

  • 49.0% mAP at 12 epochs: DINO improves performance and training convergence, with good results on small objects.The comparison concerns DETR-based transformers on COCO minival.Source: §6 · p. 17, Table 3 · p. 18
  • 35.3% mAP at 50 epochs and 44.9% at 500 epochs: original DETR's reported scores contrast with DINO's highest mAP across the evaluated epoch values.The analysis uses detection transformers with a ResNet-50 backbone.Source: §6 · p. 17
  • Deformable-DETR's deformable attention is described as capturing spatial context, while DINO's denoising learning approach is associated with faster, more effective training.The review also describes DINO's approach as learning more generalized features useful for object detection.Source: §6 · p. 19
  • YOLOS-DETR's use of DeiTsmall instead of DeiT-Ti increases model size by 20x, while REGO-DETR is nearly double DINO's size despite comparable mAP.The review links more complex architectures to increased parameter counts and overall network size.Source: §6 · p. 19
  • 26.4% mAP at 50 epochs for Deformable-DETR versus 15.2% for DETR on small objects.The review attributes Deformable-DETR's small-object performance to attention that interpolates features from neighboring pixels.Source: §6 · p. 19

7 OPEN CHALLENGES & FUTURE DIRECTIONS

The review identifies attention, backbones, and object queries as areas for further detection-transformer development, noting uneven attention to these components and fixed query counts.

7 OPEN CHALLENGES & FUTURE DIRECTIONS

  • Attention mechanisms account for 60% of DETR modifications, while backbone modifications account for 10%.The review proposes more refined attention for spatial information or task-specific constraints, and dynamic backbones that adapt complexity to image characteristics.Source: §7 · p. 19
  • DETR typically uses a fixed number of object queries despite variation in the size or number of objects across images.The review points to adjusting query counts to image content and improving query quality as future directions.Source: §7 · p. 20
  • DAB-DETR uses dynamic anchor boxes as queries, DN-DETR adds positive noise for denoising training, and DINO adds both positive and negative noise.The review presents these query-design changes as affecting detection-transformer performance.Source: §7 · p. 20

8 CONCLUSION

The review surveys recent DETR advancements and compares their performance and architectural design. It emphasizes attention and query changes for convergence and small-object performance, and calls for further work on existing challenges.

8 CONCLUSION

  • The review covers recent DETR advancements, including backbone modification, query design, and attention refinement.It compares detection transformers by performance and architectural design.Source: §8 · p. 20
  • The review describes modifications to the encoder-decoder attention module as targeting training convergence, and updated decoder queries as targeting small-object performance.Source: §8 · p. 20
  • The authors hope the review will encourage research addressing existing challenges in applying transformers to object detection.Source: §8 · p. 20

Related papers

Loading 2306.04670v3…