Source-linked AI summary
Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR
Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, Lionel M. Ni
TL;DR
DETR’s multi-scale encoder faces high computation from the many tokens in low-level features. Lite DETR interleaves high- and low-level feature updates and uses key-aware deformable attention. The paper reports a 60% computational-cost reduction while retaining 99% of original performance.
Problem
Multi-scale DETR encoders incur high computation from excessive low-level feature tokens, despite multi-scale features’ importance for object detection.
Source: §1 · p. 1Method
Lite DETR interleaves updates to high- and low-level features and uses key-aware deformable attention for cross-scale fusion.
Source: §1 · p. 2Results
Lite DETR reduces computational cost by 60% while keeping 99% of the original performance.
Source: §5 · p. 9Takeaways & Limitations
The efficient encoder design can be plugged into other DETR-based models, where it reduces encoder GFLOPs by 62%–78% while keeping 99% of original performance.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Recent DEtection TRansformer-based (DETR) models have obtained remarkable performance. Its success cannot be achieved without the re-introduction of multi-scale feature fusion in the encoder. However, the excessively increased tokens in multi-scale features, especially for about 75\% of low-level features, are quite computationally inefficient, which hinders real applications of DETR models. In this paper, we present Lite DETR, a simple yet efficient end-to-end object detection framework that can effectively reduce the GFLOPs of the detection head by 60\% while keeping 99\% of the original performance. Specifically, we design an efficient encoder block to update high-level features (corresponding to small-resolution feature maps) and low-level features (corresponding to large-resolution feature maps) in an interleaved way. In addition, to better fuse cross-scale features, we develop a key-aware deformable attention to predict more reliable attention weights. Comprehensive experiments validate the effectiveness and efficiency of the proposed Lite DETR, and the efficient encoder strategy can generalize well across existing DETR-based models. The code will be available in \url{https://github.com/IDEA-Research/Lite-DETR}.
1. Introduction
DETR encoders need multi-scale features for object detection, but adding high-resolution features greatly increases token count and computation. Lite DETR interleaves updates to high- and low-level features and adds key-aware deformable attention; experiments report lower computation while retaining detection performance.
1. Introduction
- Adding a C3 feature at 1/8 scale yields 16 times as many tokens as C5 at 1/32 scale, making self-attention 256 times more costly.The paper identifies multi-scale token growth as a computational challenge for DETR encoders.Source: §1 · pp. 1–2
- Deformable DETR reduces self-attention complexity from quadratic to linear by comparing each query with a fixed number of sampling points, enabling multi-scale features.The paper notes that its multi-scale encoder still has high computation because it introduces many query tokens.Source: §1 · p. 2
- Low-level features account for more than 75% of multi-scale tokens, while dropping them in DINO-3scale mainly reduces small-object AP by 10% and has little impact on large-object AP.This motivates reducing low-level computation without losing the local details important for small objects.Source: §1 · p. 2
- Lite DETR updates high-level and low-level features in an interleaved encoder block, first querying all scales with high-level features and updating low-level features later.The strategy reduces query tokens to 5%–25% of the original while retaining multi-scale features.Source: §1 · p. 2
- Key-aware deformable attention samples keys and values at shared locations, using the keys to predict more reliable attention weights.The authors report that it helps recover performance with the efficient encoder block.Source: §1 · p. 2
- 60% fewer detection-head GFLOPs with 99% of original detection performance; Lite-DINO-SwinT reaches 53.9 AP with 159 GFLOPs.The introduction reports these as results of comprehensive experiments.Source: §1 · p. 2
2. Related Work
Related work describes DETR’s encoder and the trade-off between multi-scale features and computation. Prior efficiency approaches reduce encoder work in different ways, while some alternatives are reported to lag behind improved deformable-encoder detectors.
2. Related Work
- Vanilla DETR uses single-scale features from the backbone’s last stage, while its encoder processes features with dense self-attention for fusion and enhancement.The encoder is compared with the feature pyramid network role in convolutional detectors.Source: §2 · pp. 2–3
- Efficient DETR reduces encoder layers using dense priors for decoder query initialization, whereas Sparse DETR sparsely updates salient tokens using a scoring network.The text notes that Sparse DETR’s multi-layer encoder detection loss can make it difficult to generalize to other DETR-based models.Source: §2 · p. 3
- DETR++ replaces the encoder with BiFPN, VIDT removes it with a stronger decoder, and IMFA samples sparse scale-adaptive features from selected areas.The paper says these models still lag behind improved detectors based on the deformable encoder.Source: §2 · p. 3
3. Method
Lite DETR reduces multi-scale encoder computation by updating high-level features more frequently than low-level features, then fusing the scales with key-aware deformable attention.
3.1. Motivation and Analysis
- Low-level features provide local details for small objects, but dropping S4 in DINO-3scale gains 48% GFLOPs efficiency at the cost of 4.9% AP and 10.2% AP on small-object detection.Low-level features account for more than 75% of tokens, while the three higher-level scales account for about 25%.Source: §3.1 · p. 3
3.2. Model Overview
- The encoder splits backbone features into high- and low-level groups, updates them at different frequencies, and uses KDA to address lagged low-level updates.The design follows multi-scale deformable DETR and uses an efficient encoder block to trade off precision and efficiency.Source: §3.2 · p. 4
3.3. Interleaved Update
- High-level features query all feature maps for cross-scale updates, while low-level features are updated less often to retain a full-scale pyramid at lower computation cost.Each block updates high-level features A times and low-level features once; the block is repeated B times.Source: §3.3 · p. 4, Figure 2 · p. 4
3.4. Iterative High-level Feature Cross-Scale Fusion
- Using the first two or three scales as high-level queries reduces queries by 94.1% or 75.3%, respectively, while high-level features extract information from all scales.The high-level features are updated iteratively, with updated features serving as queries in subsequent layers.Source: §3.4 · pp. 4–5
3.5. Efficient Low-level Feature Cross-Scale Fusion
- Original low-level features query updated high-level features alongside the original low-level features; the output concatenates the updated features into a multi-scale representation.Source: §3.5 · p. 5
3.6. Key-aware Deformable Attention
- KDA samples both keys and values for each query, then applies scaled dot-product attention to predict more reliable weights across scales at the same complexity as deformable attention.Interleaving makes it difficult for queries to predict weights and sampling locations across asynchronous feature maps.Source: §3.6 · pp. 5–6
3.7. Discussion with Sparse DETR and other Efficient Variants
- Sparse DETR token selection may be hard to generalize and relies on additional components; on DINO with ResNet-50, adding encoder detection loss alone causes a 1.4 AP drop.The discussion also notes that this auxiliary detection supervision can make the encoder harder to apply to existing models.Source: §3.7 · p. 6
4. Experiments
Experiments evaluate Lite DETR on COCO and across several DETR-based models, reporting lower encoder computation with retained performance. Ablations and attention visualizations examine component effects and KDA behavior.
4.1. Setup
- Experiments use COCO 2017 val2017 AP and replace the encoders of Deformable DETR, H-DETR, and DINO while keeping their other model components unchanged.KDA uses M=8 and K=4 for comparison with deformable attention.Source: §4.1 · p. 6
4.3. Efficiency Improvements on Other DETR-based Models
- Encoder GFLOPs fall by 78%–62% while retaining 99% of original performance; Lite-DINO on Swin-Tiny reaches 53.9 AP at 159 GFLOPs and outperforms YOLO at matched GFLOPs.Table 4 compares efficient DETR-based and CNN-based models; the reported Lite-DINO result is based on Swin-Tiny.Source: Table 4 · p. 7, §4.3 · pp. 7–8
4.2. Efficiency Improvements on Deformable DETR
- Lite-Deformable DETR achieves comparable performance to Deformable DETR with around 40% of its original encoder GFLOPs, while high-level fusion reduces encoder-layer queries to 5%–25% of original tokens.It also outperforms Sparse DETR-rho-0.3 by 0.7 AP with fewer GFLOPs.Source: §4.2 · p. 7
4.4. Visualization of KDA
- Figure 5 contrasts deformable and KDA sampling; KDA is reported to produce more reliable weights on high-resolution maps and focus better on meaningful regions of S4.The visualization shows sampling locations for an S1 object-center query and top-weight locations across S3 and S4.Source: Figure 5 · p. 8, §4.4 · p. 8
4.5. Ablation Studies
- Each component adds small computational cost while improving performance, and efficient DINO-3scale has AP_S comparable to original DINO-4scale; increasing modules to (2 + 1) × 4 brings no further improvement.The ablations examine component effects and vary high-level scales, encoder blocks, and iterative high-level fusion.Source: Table 5 · p. 8, Table 6 · p. 8, §4.5 · p. 8
5. Conclusion
Lite DETR updates high- and low-level features at different frequencies with cross-scale fusion and key-aware deformable attention. It reports reduced computation while retaining performance, and identifies runtime implementation optimization as future work.
5. Conclusion
- Lite DETR reduces computational cost by 60% while keeping 99% of original performance.The conclusion attributes this result to the proposed efficient encoder.Source: §5 · p. 9
- The encoder updates high- and low-level features at different frequencies through cross-scale fusion, with key-aware deformable attention intended to restore small-object performance.The conclusion describes the design as a computational-cost and precision trade-off.Source: §5 · p. 9
- The efficient encoder is designed to plug into multiple DETR-based detection models and is presented as a simple baseline for resource-constrained applications.The paper states this as the design's intended scope and hoped-for use.Source: §5 · p. 9
- The work focuses on computational complexity and leaves runtime implementation optimization for future work.Source: §5 · p. 9