Less is More: Focus Attention for Efficient DETR

Dehua Zheng; Wenhui Dong; Hailin Hu; Xinghao Chen; Yunhe Wang

Source-linked AI summary

Less is More: Focus Attention for Efficient DETR

Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, Yunhe Wang

arXiv:2307.12612v1Jul 24, 2023cs.CVcs.AI

TL;DR

DETR-like models face redundant encoder computation, while sparse methods can rely on unreliable token-selection statistics and discard useful object information. Focus-DETR scores multi-scale tokens using localization and category semantics, then combines token pruning with dual-attention enhancement. It achieves 50.4AP (+2.2) over Sparse DETR on COCO at comparable complexity, while reducing DINO computation by 45% with only a 0.5 AP loss.

Problem
DETR encoders perform redundant computation across tokens, while existing sparse methods can select unreliable tokens and harm detection performance.
Method
Focus-DETR uses multi-scale scoring that combines localization and category semantics, abandons background tokens, and enhances fine-grained object queries with dual attention.
Results
50.4AP (+2.2) on COCO is achieved over Sparse DETR at comparable complexity, while applying Focus-DETR to DINO reduces computation by 45% with only a 0.5 AP loss.
Takeaways & Limitations
Focus-DETR provides a better computation–accuracy trade-off by precisely selecting foreground and fine-grained tokens for enhancement.
Takeaways & Limitations
More hierarchical semantic grading, including object boundaries or centers, remains to be explored, and dual attention depends critically on precise scoring.

Abstract

from arXiv · show

DETR-like models have significantly boosted the performance of detectors and even outperformed classical convolutional models. However, all tokens are treated equally without discrimination brings a redundant computational burden in the traditional encoder structure. The recent sparsification strategies exploit a subset of informative tokens to reduce attention complexity maintaining performance through the sparse encoder. But these methods tend to rely on unreliable model statistics. Moreover, simply reducing the token population hinders the detection performance to a large extent, limiting the application of these sparse models. We propose Focus-DETR, which focuses attention on more informative tokens for a better trade-off between computation efficiency and model accuracy. Specifically, we reconstruct the encoder with dual attention, which includes a token scoring mechanism that considers both localization and category semantic information of the objects from multi-scale feature maps. We efficiently abandon the background queries and enhance the semantic interaction of the fine-grained object queries based on the scores. Compared with the state-of-the-art sparse DETR-like detectors under the same setting, our Focus-DETR gets comparable complexity while achieving 50.4AP (+2.2) on COCO. The code is available at https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR and https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR.

1. Introduction

DETR-like detectors improve accuracy but incur redundant encoder computation because conventional attention treats tokens uniformly, while existing sparse selection can miss important object tokens. Focus-DETR addresses this with dual attention and scoring that combines localization and category semantics, achieving comparable complexity with stronger detection results.

Motivation: 8.8× greater encoder than decoder calculation cost in Deformable DETR exposes the encoder as the main computational bottleneck.The comparison motivates reducing redundant encoder attention rather than focusing only on the decoder.
Limitations of Existing Sparsity: Sparse DETR preserves only 30% of encoder queries, but its decoder-attention-based scoring can select noisy tokens and overlook necessary object tokens.Its foreground supervision depends on decoder cross-attention maps derived from encoder priors, and performance can decay with learnable queries.
Approach: Focus-DETR reconstructs the encoder with dual attention and a scoring mechanism that combines localization and category semantic information from multi-scale features.The method progressively identifies foreground tokens and focuses enhancement on fine-grained object tokens.
Results: 50.4AP (+2.2) on COCO is achieved by Focus-DETR over Sparse DETR under the same setting with comparable computational complexity.The method is designed as a more accurate sparse DETR-like detector.

2. Related work

Related work reduces Transformer detection cost through architectural simplification, sparse token selection, and broader dynamic-token strategies. These approaches generally remove redundant tokens or computation while preserving detection performance, but sparse DETR methods remain limited by suboptimal selection and multi-scale utilization.

Transformer-based Detectors: Transformer-based detectors improve convergence and detection precision through architectural and training advances built around backbone, encoder, decoder, and matching components.DETR formulates detection as set prediction and uses Hungarian matching for supervision.
Lightweight Vision Transformers: Vision Transformer efficiency methods dynamically adjust token populations, preserve spatial structure, or slim patches layer by layer to lower computational complexity.The shared strategy is to abandon redundant tokens while retaining useful features.
Efficient DETR Detectors: Efficient DETR reduces encoder and decoder depth, while PnP-DETR and Sparse DETR remove weakly semantic background tokens to reduce redundant computation.These methods target efficiency while aiming to preserve performance comparable to DETR or Deformable DETR.

3. Methodology

Focus-DETR selects foreground and fine-grained object tokens using multi-scale localization and semantic scores, then processes them with dual attention to improve information mixing efficiently.

Scoring mechanism: Focus-DETR constructs a scoring mechanism that discriminates foreground and fine-grained object tokens using localization and category semantic information from multi-scale features.The method uses a foreground token selector followed by a multi-category score predictor.
Scoring mechanism: Foreground selection is supervised by ground-truth boxes and labels, avoiding Sparse DETR’s reliance on decoder cross-attention maps and improving token-selection reliability.The label assignment uses feature-map scale ranges with 50% overlap between adjacent intervals to strengthen boundary predictions.
Scoring mechanism: Top-down score modulation transfers high-level foreground scores through upsampling to complement lower-level feature maps and correlate localization information across scales.Scores are predicted for each feature map and propagated layer by layer with learnable modulation coefficients.
Scoring mechanism: The multi-category predictor ranks selected foreground tokens by the product of foreground and category scores, excluding background categories from enhanced attention.The resulting score determines which fine-grained object tokens participate in attention computation.
Encoder with dual attention: Dual attention first enhances fine-grained object-token interactions, then scatters the enhanced tokens back to foreground queries to enrich their semantics at negligible additional cost.This design addresses limited distant-token mixing while preserving the original foreground-token structure.
Complexity analysis: With a preserved foreground-token ratio of 0.3, the Transformer calculation cost decreases by over 60% under the stated common setting.The analysis uses K=4, C=256, N_qd=900, and compares the encoder query ratio γ.

4. Experiments

Experiments on COCO evaluate Focus-DETR’s accuracy–efficiency trade-off, backbone adaptability, comparisons with sparse baselines, and the contributions of its token-selection and dual-attention components.

4.2. Main Results: Focus-DETR achieves state-of-the-art AP–GFLOPs trade-offs among the compared DETR-like detectors.The comparison plots average precision against GFLOPs, with GFLOPs measured using 100 validation images.
4.2. Main Results: With ResNet-101, Focus-DETR achieves higher AP with fewer GFLOPs than the compared DETR-like models.Experiments also report excellent performance with an ImageNet-pretrained Swin Transformer backbone.
4.3. Extensive Comparison: Across three DETR variants, Sparse DETR reduces AP by 1.9, 1.2, and 2.7, whereas Focus-DETR gains 1.7, 1.1, and 2.2 AP with similar latency.The corresponding low correlations between decoder attention maps and selected foreground tokens indicate ineffective foreground selection in those Sparse DETR integrations.
4.4. Ablation Studies: Supervised foreground scoring, top-down multi-scale modulation, cascade selection, and dual attention improve performance by +1.0, +0.4, +0.5, and +0.8 AP, respectively.Multi-scale score prediction independently yields consistent gains of +0.5 or +0.7 AP, while top-down guidance is 0.2 AP better than bottom-down guidance.
4.4. Ablation Studies: At 128 GFLOPs, Focus-DETR exceeds Sparse DETR by +2.7 AP and DINO with Sparse DETR’s strategy by +1.4 AP at similar computation cost.The paper attributes the gains to its foreground scoring and fine-grained feature enhancement.

5. Conclusion

Focus-DETR targets a better efficiency–accuracy trade-off by selecting informative foreground and fine-grained tokens through multi-level semantic discrimination.

Focus-DETR precisely selects foreground and fine-grained tokens for enhancement, improving the computation-efficiency and model-accuracy trade-off.Its scoring mechanism uses both positional and semantic information, and experiments report state-of-the-art token pruning performance for DETR-like models.

A.1.1 Cascade Structure

Focus-DETR uses a cascade structure that progressively reduces the foreground region scope while preserving comparable computational complexity across keep-ratio designs.

A.1.1 Cascade Structure: Deformable attention complexity is linear in preserved-token count, so even and cascade keep-ratio structures have no significant complexity difference.The compared examples are {0.4,0.4,0.4,0.4,0.4,0.4} and {0.65,0.55,0.45,0.35,0.25,0.15}.
A.1.1 Cascade Structure: The cascade design gradually reduces the scope of foreground regions to increase model fault tolerance.

A.1.2 Label Assignment

Focus-DETR allows adjacent feature-scale label ranges to overlap, increasing foreground samples while preserving multi-scale prediction of object heterogeneity.

A.1.2 Label Assignment: Overlapping adjacent scale intervals improve boundary-near predictions by increasing foreground samples while preserving object heterogeneity across feature maps.Interval boundaries are assigned as integer powers of two.

A.2.1 Using Swin Transformer as the Backbone

Focus-DETR remains effective with Swin Transformer backbones and across training settings, while visualizations show progressively refined foreground selection and scale-aware token retention.

A.2.1 Using Swin Transformer as the Backbone: 56.0AP with Swin-B-224-22K and 55.9AP with Swin-B-384-22K demonstrate strong Focus-DETR performance with Swin Transformer backbones.Using Swin-T, Focus-DETR reaches 51.9 AP and significantly improves over Deformable DETR and Sparse DETR.
A.2.1 Using Swin Transformer as the Backbone: Focus-DETR outperforms DINO at 12 epochs with ResNet50 and reaches its optimal training state at 24 epochs.The reported behavior is attributed to its foreground selection and fine-grained feature enhancement.
A.2.1 Using Swin Transformer as the Backbone: Dual attention adds +0.3 AP in two-stage Deformable DETR, 0.0 AP without two-stage, and -0.1 AP in DN-Deformable-DETR.The experiment applies dual attention to models equipped with Sparse DETR and compares them with Focus-DETR.
A.2.1 Using Swin Transformer as the Backbone: Layer-6 captures more accurate foreground regions with fewer tokens, while multi-scale maps retain tokens according to different object scales.The visualizations compare Focus-DETR with Sparse DETR and show progressively refined foreground areas across encoder layers.