PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Object Detection with Adaptive Clustering Transformer

Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang, Hongsheng Li, Hao Dong

arXiv:2011.09315v2cs.CV

TL;DR

DETR’s high-resolution inputs make its encoder quadratically costly, while efficient transformer variants can require costly architectural changes and trial-and-error. This paper introduces ACT, a drop-in adaptive query-clustering module that approximates query-key interaction; it reduces DETR FLOPs from 73.4 Gflops to 58.2 Gflops with 0.7% AP loss, which MTKD further reduces to 0.2%.

  • Problem

    DETR’s high-resolution inputs make its encoder quadratically costly, while efficient transformer variants can require architectural changes and costly trial-and-error due to slow convergence.

    Source: §1 · p. 2

  • Method

    ACT replaces the DETR encoder transformer with an adaptive query-clustering module that approximates query-key interactions using prototypes and does not require retraining.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    73.4 Gflops to 58.2 Gflops (excluding Backbone Resnet FLOPs) with no training and 0.7% AP loss; MTKD further reduces AP loss to 0.2%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ACT reduces DETR’s computation and memory costs without retraining, using its compatibility with the original Transformer.

    Source: §6 · p. 10

    The cited discussion identifies a limitation of K-means, not ACT: fixed cluster counts may yield poor estimates or wasted computation when query distributions vary.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

End-to-end Object Detection with Transformer (DETR)proposes to perform object detection with Transformer and achieve comparable performance with two-stage object detection like Faster-RCNN. However, DETR needs huge computational resources for training and inference due to the high-resolution spatial input. In this paper, a novel variant of transformer named Adaptive Clustering Transformer(ACT) has been proposed to reduce the computation cost for high-resolution input. ACT cluster the query features adaptively using Locality Sensitive Hashing (LSH) and ap-proximate the query-key interaction using the prototype-key interaction. ACT can reduce the quadratic O(N2) complexity inside self-attention into O(NK) where K is the number of prototypes in each layer. ACT can be a drop-in module replacing the original self-attention module without any training. ACT achieves a good balance between accuracy and computation cost (FLOPs). The code is available as supplementary for the ease of experiment replication and verification. Code is released at \url{https://github.com/gaopengcuhk/SMCA-DETR/}

1 CFCS, CS Dept., Peking University · 1 Introduction

ACT reduces DETR’s computation by clustering similar query features and calculating attention for representative prototypes, while adapting cluster counts to feature distributions. It can replace the original transformer without retraining, and MTKD can further reduce the accuracy gap and enable switching between models at inference.

1 Introduction

  • DETR [3] simplifies detection with end-to-end set prediction, but high-resolution inputs make its encoder computationally expensive.Unlike two-stage detection pipelines, DETR avoids anchors and region proposals; its encoder has quadratic all-pairs interaction across positions.Source: §1 · p. 2
  • Prior transformer variants reduce computation but change the architecture, imposing trial-and-error costs amid DETR’s slow convergence, reported as 1920 GPU hours on one V100.The paper frames the goal as improving DETR’s performance-computation trade-off with acceptable computing resources.Source: §1 · p. 2
  • ACT is a drop-in replacement for the original DETR transformer and requires no retraining; MTKD enables inference-time switching between models with different FLOPs and accuracy.MTKD guides training of ACT and the original transformer in parallel, further closing their accuracy gap.Source: §1 · pp. 2–3
  • Semantically similar, spatially close queries produce similar attention maps, so ACT clusters query features and computes attention for representative prototypes.Prototype updates are broadcast to neighboring queries according to Euclidean distance in query-feature space.Source: §1 · pp. 2–3
  • Because encoder features become more similar with depth and their distributions vary across inputs and layers, ACT adapts prototype counts using multi-round E2LSH.The paper motivates distribution-aware clustering rather than using a fixed number of cluster centers.Source: §1 · p. 3
  • 73.4 Gflops falls to 58.2 Gflops without training, with 0.7% AP loss; MTKD reduces the AP loss to 0.2%.The FLOPS figures exclude Backbone Resnet FLOPs.Source: §1 · p. 3

2 Related Work

Prior efficient-transformer approaches modify the original structure, whereas ACT retains it and reduces computation without retraining. In object detection, ACT targets DETR’s inference cost, unlike Deformable DETR and SMCA, which mainly improve training.

2.1 Review of Attention Model on NLP and CV

  • Transformer exchanges information between all entity pairs and has achieved strong results across tasks, but its quadratic sequence-length complexity makes scaling difficult.The paper frames efficient transformer modifications as responses to this computational bottleneck.Source: §2.1 · p. 4
  • Reformer [23], Performer [6], Linear Attention [21], Progressive Elimination [16], Asymmetric Attention [47], and Global Graph Reasoning [5] reduce transformer computation through clustering, approximation, elimination, or compact representations.The cited approaches respectively use LSH clustering, random-feature approximation, associative multiplication, progressive input elimination, pooled key vectors, and pooled global vectors.Source: §2.1 · p. 4
  • ACT preserves the original transformer structure and reduces computation without retraining; a few epochs of knowledge-distillation fine-tuning can further narrow its performance gap.The related-work discussion contrasts ACT with earlier methods that modify the transformer and require substantial training and inference resources.Source: §2.1 · p. 4

2.2 Object Detection using Deep Learning

  • Object detection moved from sliding-window classification and one- or two-stage CNN detectors toward permutation-invariant set prediction, addressing prior pipeline burdens such as NMS, imbalanced loss, and hand-crafted anchors.The passage identifies RCNN variants as two-stage methods, YOLO and SSD as one-stage methods, and describes end-to-end people detection as an earlier set-prediction approach.Source: §2.2 · p. 4
  • DETR made set-prediction performance comparable to Faster RCNN and simplified detection, but slow convergence and quadratic self-attention inference cost motivated variants; Deformable DETR and SMCA improve convergence, while ACT targets inference cost without retraining.Deformable DETR uses learnable sparse sampling with a multi-scale deformable encoder, while SMCA introduces a Gaussian prior in the transformer decoder.Source: §2.2 · p. 5

3 Adaptive Clustering Transformer

The section introduces DETR’s encoder–decoder structure, then presents ACT’s LSH-based query clustering and prototype-based attention approximation. It also describes MTKD, which trains ACT alongside the original transformer to support switching between them.

3.1 Main Structure of DETR

  • DETR uses a six-layer encoder and decoder, with learned object queries in the decoder and shared feed-forward prediction heads.The encoder processes flattened, position-encoded ResNet features; each encoder and decoder layer has eight-head self-attention, while decoder layers also include eight-head co-attention.Source: §3.1 · p. 5

3.2 Adaptive Clustering Transformer

  • E2LSH groups nearby queries into hash buckets, and each cluster’s prototype is the center of its queries.The method controls the hash function and number of rounds so vectors within distance ε share a bucket with probability greater than p.Source: §3.2 · pp. 5–6
  • ACT replaces query-key attention with prototype-key interactions, reducing complexity from O(NMDk + NMDv) to O(NLDK + CMDk + CMDv).Here, C is the number of prototypes and L the number of hash rounds; both are much smaller than N and M. Prototype attention outputs are broadcast to the original queries.Source: §3.2 · p. 6

3.3 Multi-Task Knowledge Distillation

  • MTKD trains ACT and the original transformer in parallel, distilling bounding-box regression to enable switching between them.It transfers regression knowledge only because that branch is more sensitive to ACT’s approximation error than classification, and the authors report faster convergence with regression-only transfer.Source: §3.3 · pp. 6–7

4 Experiment

Experiments evaluate ACT on COCO 2017 detection and examine its approximation settings, clustering targets, accuracy–computation tradeoffs, and real-world inference costs.

4.1 Dataset

  • Experiments use COCO 2017’s 118k training and 5k validation images, report bbox AP, and average FLOPs over the first 100 validation images.The FLOPs accounting includes convolutional and fully connected layers, attention matrix operations, E2LSH, and clustering.Source: §4.1 · p. 7

4.2 Experiment Setup

  • The baseline is pre-trained DETR-DC5 with a dilated ResNet-50 backbone and six encoder and six decoder layers; ACT replaces encoder attention while other parts remain unchanged.The study selects r using attention-map mean square error on 1,000 training images and varies L to control FLOPs.Source: §4.2 · pp. 7–8

4.3 Ablation Study

  • Estimation error falls as L increases and r decreases; above r = 6, further increases have little effect, while r ≤ 6 makes lower error costly in FLOPs, so later experiments fix r = 8.The study compares r values with L = 24 and L values with r = 8 using attention-map mean square error on 1,000 training images.Source: §4.3 · p. 8
  • With similar FLOPs, clustering queries, keys, or both yields similar AP; clustering keys can improve efficiency when keys substantially outnumber queries.The comparison adjusts L to bring the three clustering choices to similar FLOPs.Source: §4.3 · p. 8, Table 2 · p. 9

4.4 Final Performance

  • ACT reduces DETR-DC5 FLOPs from 184.1 GFLOPs to 168.9 GFLOPs with only 0.7% AP loss, and exceeds K-means AP under equalized budgets.The comparison with DETR-DC5 and K-means is reported in the speed-accuracy analysis.Source: §4.4 · p. 8
  • With MTKD, ACT at L = 16 gains 4.3% AP, while ACT at L = 32 achieves almost DETR-DC5’s performance.ACT at L = 32 also approaches Faster RCNN-DC5 performance with much fewer FLOPs; its APL matches DETR-DC5, and ACT’s APL is 0.3% higher than DETR’s.Source: §4.4 · pp. 9–10
  • ACT adaptively selects prototype counts, avoiding K-means’ fixed cluster count across layers, sensitivity to C, and possible nonconvergence or empty clusters.The passage attributes the fixed-count difficulty to variation in query distributions across images and encoder layers.Source: §4.4 · p. 9
  • ACT’s real-world acceleration is consistent with its FLOPs analysis, and it significantly reduces memory cost on a GTX TITAN X at batch size 1.The measurements are described as inference time and memory cost; no numeric values are supplied in the passage.Source: §4.4 · p. 10

5 Visualisation of Adaptive Clustering

Visualizations show that ACT clusters queries by semantics and location, while adaptively using fewer prototypes in deeper encoder layers as features become more redundant.

5 Visualisation of Adaptive Clustering

  • Figure 6 shows clusters corresponding to cow, sky, and field features, indicating that queries with similar semantics and locations tend to group together.Source: §5 · p. 10
  • Figure 7 shows the prototype-to-query ratio trending downward in deeper encoder layers, where features are more redundant.Source: §5 · p. 10

6 Conclusion

The paper proposes Adaptive Clustering Transformer (ACT) to reduce object-detection computation and memory costs by adaptively clustering redundancy in pre-trained DETR, without retraining. Future work will explore training ACT from scratch and applying it to cross-scale information fusion over multi-scale FPN [27].

6 Conclusion

  • ACT reduces computation and memory costs for object detection by adaptively clustering redundancy in pre-trained DETR without retraining.The authors attribute the lack of retraining to ACT's compatibility with Transformer.Source: §6 · p. 10
  • DETR training requires 500 epochs and approximately 1920 GPU hours on a single V100 GPU, while most previous efficient transformers require retraining when applied to DETR.Source: §6 · p. 10
  • Future work will investigate training ACT from scratch and using it for cross-scale information fusion over multi-scale Feature Pyramid Network (FPN) [27].Source: §6 · p. 10

Related papers

Loading 2011.09315v2…