PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Convergence of DETR with Spatially Modulated Co-Attention

Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, Hongsheng Li

arXiv:2101.07448v1cs.CV

TL;DR

DETR’s 500-epoch training requirement reflects slow convergence and limits iteration on object-detection designs. SMCA replaces decoder co-attention with regression-aware spatial modulation, using predicted box centers and scales plus multi-scale and multi-head designs. Full SMCA reaches 45.6 mAP at 108 epochs versus DETR-DC5’s 43.3 mAP at 500 epochs.

  • Problem

    DETR converges slowly, requiring 500 rather than 40 epochs compared with other object detectors, which increases design-cycle time and hinders broader usage.

  • Method

    SMCA replaces DETR’s decoder co-attention with a dynamic Gaussian-like spatial weight map centered and scaled around each query’s initial box estimate.

  • Results

    45.6 mAP at 108 epochs versus 43.3 mAP at 500 epochs is reported for full SMCA and DETR-DC5, respectively.

  • Takeaways & Limitations

    SMCA shows that DETR can converge faster while exploring global information and changing only the decoder’s co-attention mechanism.

Abstract

from arXiv · show

The recently proposed Detection Transformer (DETR) model successfully applies Transformer to objects detection and achieves comparable performance with two-stage object detection frameworks, such as Faster-RCNN. However, DETR suffers from its slow convergence. Training DETR \cite{carion2020end} from scratch needs 500 epochs to achieve a high accuracy. To accelerate its convergence, we propose a simple yet effective scheme for improving the DETR framework, namely Spatially Modulated Co-Attention (SMCA) mechanism. The core idea of SMCA is to conduct regression-aware co-attention in DETR by constraining co-attention responses to be high near initially estimated bounding box locations. Our proposed SMCA increases DETR's convergence speed by replacing the original co-attention mechanism in the decoder while keeping other operations in DETR unchanged. Furthermore, by integrating multi-head and scale-selection attention designs into SMCA, our fully-fledged SMCA can achieve better performance compared to DETR with a dilated convolution-based backbone (45.6 mAP at 108 epochs vs. 43.3 mAP at 500 epochs). We perform extensive ablation studies on COCO dataset to validate the effectiveness of the proposed SMCA.

1. Introduction

DETR simplifies object detection but converges slowly because its decoder must learn which visual regions correspond to each query. SMCA replaces decoder co-attention with location-constrained modulation, and its multi-scale, multi-head version reaches higher mAP in far fewer epochs.

  • DETR removes hand-crafted anchors and NMS but requires 500 rather than 40 epochs, slowing algorithm design and broader usage.
  • DETR queries may attend to visual regions unrelated to their predicted boxes, forcing the decoder to search for suitable regions during training.
  • SMCA replaces DETR’s decoder co-attention with a plug-and-play, location-constrained regression module that predicts spatial priors around initial box estimates.
  • SMCA uses a dynamic Gaussian-like spatial prior while retaining global information, differing from approaches that accelerate DETR through sparse or deformable local sampling.
  • 41.0 mAP at 50 epochs and 42.7 mAP at 108 epochs are achieved by basic SMCA without multi-scale features or multi-head attention.
  • 45.6 mAP at 108 epochs versus 43.3 mAP at 500 epochs is achieved by full SMCA with multi-scale features and multi-head spatial modulation.
  • Ablation studies on the COCO 2017 dataset evaluate the SMCA module and network design.

2. Related Work

Object detection research spans two-stage, one-stage, and end-to-end methods, while Transformer architectures introduce global pairwise information exchange. Related work addresses DETR’s slow convergence, multi-scale representation, Transformer efficiency, and dynamic modulation.

  • Two-stage detectors generate region proposals before ROI feature extraction and classification, whereas one-stage detectors classify and localize directly over dense windows.
  • End-to-end detectors directly supervise bounding boxes with Hungarian bipartite matching, but DETR converges more slowly than one-stage and two-stage detectors.
  • Deformable DETR accelerates convergence through learnable sparse sampling and a multi-scale deformable encoder, while TSP identifies co-attention and bipartite matching as major causes of DETR’s slow convergence.
  • Focal loss reduces gradients from negative samples, while Feature Pyramid Network variants provide multi-scale features for object detection.
  • Transformer exchanges information between all input pairs through key-query-value attention but has quadratic complexity for longer sequences.
  • Reformer, Linformer, FastTransformer, and LongFormer reduce Transformer computation through reversibility, clustering, altered multiplication, or local-window attention.
  • Dynamic modulation controls information flow through predicted functions such as LSTM gates or attention maps used in image captioning.

3. Spatially Modulated Co-Attention

SMCA modifies DETR’s decoder co-attention with spatial priors predicted from each object query, while extending the architecture with multi-head and multi-scale attention designs. These components constrain attention toward likely object regions and support scale-specific feature selection.

  • A Revisit of DETR: DETR encodes CNN visual features with self-attention, then object queries co-attend to the encoded spatial features before predicting classes and bounding boxes.Hungarian matching identifies learning targets for the object queries.
  • Dynamic spatial weight maps: SMCA predicts each query’s initial object center and scale, generating a Gaussian-like spatial weight map over image locations.The center is predicted in normalized coordinates and unnormalized to image coordinates; width and height scales are estimated separately.
  • Spatially Modulated Co-Attention: SMCA adds the logarithm of the spatial map to dot-product co-attention logits before softmax, increasing weights near predicted box locations.The bandwidth parameter β controls the Gaussian-like map’s spatial range, with an initially broad map used to provide informative gradients.
  • Multi-head spatial modulation: The basic SMCA shares one spatial map across attention heads, whereas the multi-head version predicts head-specific centers and scales to emphasize diverse context.Head-specific spatial maps are intended to improve detection accuracy by modulating the co-attention maps differently.
  • Multi-scale visual features: SMCA incorporates multi-scale visual features from backbone outputs with downsampling rates 16, 32, and 64, together with scale-selection attention.This avoids naively rescaling and concatenating all scales because some queries may require information from a specific scale.

4. Experiments

Experiments on COCO 2017 evaluate SMCA through implementation comparisons, ablations, visualization, and comparisons with DETR-like detectors. The results show faster convergence and improved detection performance, with marginal computational overhead.

  • Implementation: SMCA adds decoder spatial modulation to DETR while keeping the architecture largely unchanged, with only marginal increases in computational cost and training time.SMCA introduces a new co-attention modulation and an extra linear network for generating the spatial prior.
  • Spatial modulation ablation: 5.4 mAP: head-shared spatial modulation improves baseline DETR from 34.8 to 40.2 mAP at 50 epochs.The comparison uses the same learning rate, training schedule, self-attention parameters, and loss coefficients.
  • Spatial modulation ablation: 41.0 mAP: multi-head spatial modulation improves over head-shared modulation at 40.2 mAP.Separate spatial weight maps allow different heads to learn diverse attention patterns simultaneously.
  • Spatial modulation ablation: 38.5, 40.4, and 41.0 mAP: fixed, single-scale, and independent height-width scales progressively improve spatial modulation.The results support predicting centers and independently adapting width and height scales to object-size variation.
  • Multi-scale ablation: 43.7 mAP: multi-scale encoding improves SMCA from 41.0 mAP, while removing scale-selection attention reduces performance to 42.6 mAP.The multi-scale encoder uses a 2Intra-Multi-2Intra self-attention design.
  • Multi-scale ablation: 43.3 mAP: replacing the 2Intra-Multi-2Intra design with five intra-scale layers lowers performance from 43.7 mAP.The authors attribute the drop to lacking cross-scale information exchange and note that all-multi-scale attention increases FLOPs.
  • Overall comparison: 45.6 AP at 109 epochs: SMCA exceeds Faster RCNN-FPN-R50 at 42.0 AP and outperforms DETR-like alternatives under reported training schedules.SMCA reaches 45.6 at 108 epochs versus 42.8 at 300 epochs for UP-DETR and 45.0 at 96 epochs for TSP-RCNN.

5. Conclusion

SMCA reduces DETR’s training schedule from 500 to 108 epochs while increasing mAP from 43.4 to 45.6 at comparable inference cost. The paper also identifies future applications beyond object detection and flexible local-global feature fusion as research directions.

  • 108 epochs reduces DETR’s original 500-epoch training schedule while mAP increases from 43.4 to 45.6 at comparable inference cost.
  • SMCA demonstrates the potential of exploring global information for high-quality object detection.
  • Future work will explore SMCA beyond object detection and flexible fusions of local and global features.
Loading 2101.07448v1…