PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity

Byungseok Roh, JaeWoong Shin, Wuhyun Shin, Saehoon Kim

arXiv:2111.14330v2cs.CVcs.LG

TL;DR

DETR-based detection remains computationally expensive because multiscale features greatly increase encoder tokens and encoder attention remains a bottleneck. Sparse DETR selectively updates tokens expected to be referenced by the decoder and applies auxiliary detection loss to selected tokens; on COCO, it outperforms Deformable DETR with 10% of encoder tokens while reducing computation and increasing FPS.

  • Problem

    Multiscale Deformable DETR improves detection but increases encoder tokens and leaves encoder attention computationally costly.

  • Method

    Sparse DETR predicts decoder-referenced encoder tokens for sparsification and applies an auxiliary detection loss only to selected tokens.

  • Results

    Sparse DETR outperforms Deformable DETR with 10% of encoder tokens, reducing overall computation by 38% and increasing FPS by 42%.

  • Takeaways & Limitations

    Encoder token sparsification preserves detection performance while lowering the computational cost of the DETR encoder.

  • Takeaways & Limitations

    The decoder-oriented Objectness Score criterion is sub-optimal because its separately selected encoder tokens are not explicitly considered by the decoder.

Abstract

from arXiv · show

DETR is the first end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efficiency on high resolution feature maps. The subsequent work, Deformable DETR, enhances the efficiency of DETR by replacing dense attention with deformable attention, which achieves 10x faster convergence and improved performance. Deformable DETR uses the multiscale feature to ameliorate performance, however, the number of encoder tokens increases by 20x compared to DETR, and the computation cost of the encoder attention remains a bottleneck. In our preliminary experiment, we observe that the detection performance hardly deteriorates even if only a part of the encoder token is updated. Inspired by this observation, we propose Sparse DETR that selectively updates only the tokens expected to be referenced by the decoder, thus help the model effectively detect objects. In addition, we show that applying an auxiliary detection loss on the selected tokens in the encoder improves the performance while minimizing computational overhead. We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder tokens on the COCO dataset. Albeit only the encoder tokens are sparsified, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR. Code is available at https://github.com/kakaobrain/sparse-detr

1 INTRODUCTION

Sparse DETR targets the encoder bottleneck in end-to-end detection by selectively updating informative tokens rather than processing all multiscale tokens equally. Its token sparsification and auxiliary encoder loss preserve or improve detection while reducing computation.

  • Motivation: Deformable DETR improves DETR efficiency with deformable attention, but multiscale inputs increase encoder token processing and leave computation costly.DETR cannot use multiscale features, while Deformable DETR's larger token set makes inference slower.
  • Motivation: 45% of encoder tokens are referenced by the decoder, and updating only decoder-preferred tokens causes just 0.1 AP degradation.These preliminary COCO validation observations motivate selectively updating a smaller token subset.
  • Method: Sparse DETR uses a learnable decoder cross-attention map predictor to select encoder tokens for self-attention.Unlike existing methods that process all tokens, the method distinguishes tokens expected to be referenced by the decoder.
  • Method: An auxiliary detection loss on selected encoder tokens improves performance, stabilizes learning, and permits stacking more encoder layers with marginal training overhead.The loss is applied only to selected tokens to minimize computational overhead.
  • Results: 48.2 AP is achieved with 38% lower total computational cost than the 48.0 AP baseline, using only 10% of encoder tokens.At this setting, transformer encoder block computation falls by approximately 82%.

2 RELATED WORK

Related work reduces transformer computation through lightweight attention and input-dependent token sparsification. Sparse DETR differs by targeting the encoder of an end-to-end object detector rather than primarily sparsifying classification backbones.

  • Efficient computation in vision transformers: Lightweight attention methods address the high time and memory complexity of Transformers, but many focus on single-scale attention.This focus limits direct extension to the multiscale features commonly required for object detection.
  • Input-dependent token sparsification: DynamicViT and IA-RED2 jointly learn token selectors for backbone networks evaluated on classification tasks.Sparse DETR instead studies token sparsification in the encoder of an end-to-end object detector.
  • Sparse DETR-based transformers: Deformable DETR sparsifies keys with learnable 2-d offsets while retaining dense queries, whereas Sparse DETR further reduces the query tokens.Both approaches target sparse computation within DETR-based transformer frameworks.
  • Auxiliary loss: DETR variants use auxiliary Hungarian matching objectives after decoder layers so each layer learns to detect the correct number of objects.This auxiliary-loss design is part of the broader use of auxiliary supervision in DETR variants.

3 APPROACH

Sparse DETR sparsifies encoder computation by selecting salient tokens expected to be referenced by the decoder, while retaining unselected tokens as keys. It combines learned saliency criteria with auxiliary encoder losses and top-k decoder query selection to reduce complexity and stabilize performance.

  • Encoder token sparsification: Sparse DETR formulates encoder token sparsification as selecting a small salient subset for refinement while passing unselected tokens through unchanged.Unselected tokens can still serve as keys when selected tokens are updated, allowing information transfer without updating every token.
  • Finding salient encoder tokens: The encoder selects the top-ρ% tokens using a scoring network that measures token saliency.The selected set is defined from backbone features according to a keeping ratio ρ.
  • Encoder token sparsification: Query sparsification reduces encoder attention from DETR’s O(N^2) to Sparse DETR’s O(SK), where S ≪ N is the number of salient queries.Deformable attention uses O(NK), whereas Sparse DETR further sparsifies the encoder queries.
  • Finding salient encoder tokens: Objectness Score selects top-ρ% tokens by class scores from an auxiliary detection head, but is sub-optimal because it does not explicitly target decoder use.The head is trained with a Hungarian loss and has the same structure as the final decoder detection head.
  • Finding salient encoder tokens: Decoder Cross-Attention Map saliency trains a scoring network to predict which encoder tokens are most referenced by the decoder.The method uses decoder cross-attention maps as pseudo-ground-truth saliency targets; optimization remains stable early in training and outperforms objectness-based selection empirically.
  • Additional components: Auxiliary detection heads with Hungarian loss on selected encoder tokens stabilize deeper encoders and improve detection performance.Sparse DETR also selects top-k encoder outputs as decoder object queries using an auxiliary head.

4 EXPERIMENTS

Sparse DETR is evaluated against object-detection baselines and through ablations of token selection, auxiliary loss, and dynamic sparsification. It maintains or improves detection performance while reducing encoder computation and supporting inference across sparsity levels.

  • Comparison with object detection baselines: Sparse DETR is compared with Faster-RCNN, DETR, Deformable DETR, PnP DETR, and Swin-T variants on COCO val2017.DAM-based keeping ratios of 10%–50% are evaluated with detection performance and inference costs.
  • Comparison with object detection baselines: On ResNet-50, keeping more than 30% of encoder tokens outperforms all baselines while minimizing computational cost.At 10% retention, Sparse DETR still exceeds most baselines except Deformable DETR+.
  • Comparison between token selection criteria: DAM-based selection outperforms OS-based selection at every tested ratio and nearly matches the non-sparse baseline at 50% retention.Random selection instead causes noticeable performance degradation.
  • Comparison between token selection criteria: DAM-based selection yields higher Corr than OS-based selection, indicating greater overlap between decoder-referred and encoder-refined tokens.The authors connect this alignment with better detection performance.
  • Effectiveness of the encoder auxiliary loss: The encoder auxiliary loss improves detection and allows performance to keep increasing when encoder depth is doubled to 12 layers.Without the auxiliary loss, training with 12 encoder layers fails; the authors attribute this to vanishing gradients from decoder cross-attention.
  • Dynamic sparsification for inference stage: A fixed-sparsity model retains satisfactory performance across varied inference keeping ratios, with only slight degradation at small ratios.Sparse DETR improves AP by 0.2 in a comparable 0.3/0.5 training/inference setting where PnP DETR drops 5.0 AP.

5 CONCLUSION

Sparse DETR sparsifies encoder tokens to reduce the encoder’s computational cost, outperforming Deformable DETR with fewer tokens. It reduces overall computation and increases FPS.

  • Sparse DETR outperforms Deformable DETR while using only 10% of the encoder tokens.
  • 38% lower overall computation and 42% higher FPS are reported compared to Deformable DETR.

A.1 IMPLEMENTATION DETAILS OF THE SCORING NETWORK

The scoring network token-wise processes inputs to produce a scalar logit for each token, but this local processing may overlook global statistics.

  • The scoring network uses four linear layers with Layer Normalization before the first layer and GELU after every layer except the last.Its hidden dimensions are 256, 128, and 64, and the final layer outputs a one-dimensional logit for BCE loss.

A.2 DAM CREATION IN DEFORMABLE ATTENTION

DAM is constructed from deformable-attention offsets, weights, and reference points using bilinear interpolation, then accumulated across decoder object queries.

  • Because deformable-attention offsets are fractional positions, DAM uses bilinear interpolation to obtain values.
  • For decoder query q, deformable attention combines attention weights, interpolated features at offset reference points, and feature values.The formulation uses offset p, weight A, reference point r, bilinear kernel G, and values v.
  • DAM values are accumulated for each feature-map location and then across every decoder object query.

A.3 ALTERNATIVE OBJECTIVES FOR DAM-BASED MODEL

The DAM-based scoring network can use regression, ranking, or BCE objectives, with BCE reported as the strongest alternative in the ablation.

  • Regression and pairwise ranking losses are proposed as alternatives for learning the relative saliency of encoder tokens.Regression predicts DAM values directly, whereas ranking emphasizes relative ordering among salient tokens.
  • BCE outperforms the regression and pairwise ranking alternatives for the DAM-based scoring network.The passage attributes this result to regression difficulty, evolving DAM values, and instability in ranking DAM elements during training.

A.4 EXPERIMENTAL DETAILS FOR DIFFERENT TOKEN SELECTION CRITERIA

The experiments compare token-selection criteria against a lower-bound model with no encoder block. Even at a 10% keeping ratio, every scoring method, including random selection, outperforms this lower bound.

  • At a 10% keeping ratio, all scoring methods, including random selection, outperform the lower-bound model with no encoder block.The lower-bound model directly passes backbone features to the decoder.

A.5 VANISHING GRADIENT PROBLEM IN THE DEEP END-TO-END DETECTORS

The appendix examines vanishing gradients in deep end-to-end detectors and finds that deeper encoders worsen early-layer gradient attenuation. Pre-LN alleviates this issue for DETR, while encoder auxiliary loss addresses it for Deformable DETR.

  • Vanishing gradient problem: Training DETR with more than nine encoder layers fails under the reproduced settings, despite performance improving with additional layers in prior observations.The reproduction used official-code defaults while changing only the number of encoder layers.
  • Vanishing gradient problem: Doubling encoder size further reduces early-layer gradient scale, whereas Pre-LN maintains gradient magnitude more evenly through the encoder.The observed attenuation may contribute to convergence failure in deeper encoders.
  • Vanishing gradient problem: Deformable DETR exhibits the same vanishing-gradient problem, and Pre-LN does not resolve it in that setting.
  • Vanishing gradient problem: Encoder auxiliary loss amplifies early-layer gradients through intermediate objectives, enabling deeper-encoder training and synergizing with token sparsification.

A.6 EXPERIMENTAL DETAILS FOR EFFECTIVENESS OF THE ENCODER AUXILIARY LOSS

These experiments evaluate encoder auxiliary loss, backbone scale, initialization, token sparsification, and selection visualizations. They show that auxiliary objectives support deeper training, while sparse selection can preserve detection performance and focus on object regions.

  • Encoder auxiliary loss: With auxiliary loss, training remains feasible for encoders deeper than nine layers, and accuracy improves.
  • Swin-B experiments: With Swin-B, the baseline reaches 52.5 AP at 2.4× parameters and 2.1× computational cost, while 40% keeping retains a +4.1 performance gap with auxiliary loss.
  • Preliminary sparsification experiment: At 50% keeping, the two-stage model reaches 47.9 AP versus 48.0 AP for the baseline without encoder auxiliary loss.
  • Preliminary sparsification experiment: The two-stage sparsification experiment nearly matches the 100% baseline when the keeping ratio approaches 45%, corresponding to the validation DAM non-zero rate.
  • Preliminary sparsification experiment: The preliminary experiment uses DAMs from a separately trained decoder, whereas the main method obtains DAMs jointly and trains a scoring network to predict binarized DAM targets.
  • Token-selection visualizations: DAM-based selection captures object boundaries and interiors more effectively than OS-based selection, which is more distracted by background edges.The analysis attributes boundary attention to regression loss and interior attention to classification loss.
Loading 2111.14330v2…