Source-linked AI summary
End-to-End Object Detection with Adaptive Clustering Transformer
Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang, Hongsheng Li, Hao Dong
TL;DR
DETR’s high-resolution encoder makes all-pairs attention computationally expensive, while architectural alternatives can require costly retraining. ACT adaptively clusters query features into prototypes and estimates attention through prototype-key interactions as a drop-in replacement. It reduces FLOPs from 73.4 Gflops to 58.2 Gflops with a 0.7% AP loss, reduced to 0.2% with MTKD.
Problem
DETR’s high-resolution encoder has quadratic all-pairs computation, and efficient transformer variants may require costly retraining.
Method
ACT adaptively clusters query features into prototypes and estimates attention through prototype-key interactions as a compatible DETR replacement.
Results
73.4 Gflops to 58.2 Gflops: ACT reduces DETR’s FLOPs without training, with a 0.7% AP loss; MTKD reduces the loss to 0.2%.
Takeaways & Limitations
ACT offers a computation–accuracy trade-off for DETR while preserving compatibility with the original transformer.
Takeaways & Limitations
The paper identifies K-means clustering as sensitive to a fixed cluster-count hyperparameter, whereas ACT determines prototype counts adaptively.
Abstract
from arXiv · showhide
End-to-end Object Detection with Transformer (DETR)proposes to perform object detection with Transformer and achieve comparable performance with two-stage object detection like Faster-RCNN. However, DETR needs huge computational resources for training and inference due to the high-resolution spatial input. In this paper, a novel variant of transformer named Adaptive Clustering Transformer(ACT) has been proposed to reduce the computation cost for high-resolution input. ACT cluster the query features adaptively using Locality Sensitive Hashing (LSH) and ap-proximate the query-key interaction using the prototype-key interaction. ACT can reduce the quadratic O(N2) complexity inside self-attention into O(NK) where K is the number of prototypes in each layer. ACT can be a drop-in module replacing the original self-attention module without any training. ACT achieves a good balance between accuracy and computation cost (FLOPs). The code is available as supplementary for the ease of experiment replication and verification. Code is released at \url{https://github.com/gaopengcuhk/SMCA-DETR/}
2 Shanghai AI Laboratory
The passage identifies the affiliation as the CUHK-SenseTime Joint Laboratory at The Chinese University of Hong Kong.
- The affiliation is the CUHK-SenseTime Joint Laboratory.
- The laboratory is associated with The Chinese University of Hong Kong.
- The passage lists this affiliation as item 3.
1 Introduction
The introduction motivates ACT as a computationally efficient, drop-in replacement for DETR’s encoder transformer. It adaptively clusters redundant query features and reports lower FLOPs with a small AP loss, further reduced by MTKD.
- Motivation: DETR’s high-resolution encoder incurs quadratic computation from all-pairs spatial interactions.
- Contribution: ACT replaces DETR’s encoder transformer as a compatible drop-in module without retraining.
- Design motivation: ACT uses representative prototypes because semantically similar, spatially close features produce redundant attention maps.
- Design motivation: ACT adaptively determines prototype counts because feature distributions vary across inputs and encoder layers.
- Results: 73.4 Gflops to 58.2 Gflops: DETR’s FLOPs decrease without training, with a 0.7% AP loss.
- Results: 0.2%: MTKD further reduces ACT’s AP loss and enables switching between models with different FLOPs and accuracy.
2 Related Work
Prior attention and object-detection methods address information exchange and detection pipelines in different ways. ACT instead preserves the original transformer structure while reducing computation without retraining.
- Attention models: Transformers exchange information between all pairs of entities and support tasks including object detection, classification, and language understanding.
- Efficient transformers: Reformer uses Locality Sensitivity Hashing, Performer uses random features, and Linear Attention uses associativity to reduce attention complexity.
- Efficient transformers: Earlier efficient-transformer variants modify the transformer structure and require substantial training and inference resources.
- ACT: ACT shares the original transformer structure, reducing computation without retraining while allowing further gap reduction through knowledge distillation.
- Object detection: Object detection methods include two-stage approaches such as Faster RCNN and one-stage approaches such as YOLO and SSD.
- DETR: DETR simplifies object detection through end-to-end set prediction using a transformer instead of recurrent components.
3 Adaptive Clustering Transformer
ACT replaces encoder self-attention with adaptive query clustering and prototype–key attention, reducing computation while preserving compatibility with the original transformer. LSH forms prototypes, broadcasts their estimated outputs to queries, and MTKD supports switching between ACT and the original transformer.
- DETR extracts spatial features, adds positional encodings, and processes them through a six-layer encoder with eight-head self-attention and FFN modules.
- LSH adaptively groups nearby query features into hash buckets, whose cluster centers serve as attention prototypes.The hash function uses parameter r and L independent hashing rounds; nearby vectors share buckets with higher probability.
- ACT computes attention between prototypes and keys, then broadcasts each prototype’s updated feature to its assigned queries.The estimated attention output approximates exact attention while retaining the original query positions.
- The approximation reduces exact attention complexity from O(NMD_k + NMD_v) to O(NLDK + CMD_k + CMD_v).C is the number of prototypes, while L is the number of hash rounds; both are much smaller than N and M.
- MTKD jointly trains the original transformer and ACT, using the original DETR loss plus an L2 knowledge-distillation loss on predicted bounding boxes.The method transfers regression knowledge because that branch is more sensitive to ACT’s approximation error than classification.
- Figure 4 evaluates mean square error against computational budget by varying L with r fixed and r with L fixed.
4 Experiment
Experiments on COCO evaluate ACT against DETR-DC5 and alternative clustering choices across accuracy, computation, inference time, and memory. ACT reduces computation with limited AP loss, while MTKD further narrows the accuracy gap.
- Experimental Setup: Experiments use COCO 2017, reporting bbox AP and average FLOPs over the first 100 validation images.The dataset contains 118k training images and 5k validation images; counted FLOPs include convolutional, fully connected, attention matrix, E2LSH, and clustering operations.
- Experimental Setup: ACT replaces encoder attention while leaving the remaining DETR-DC5 components unchanged.The baseline uses a ResNet-50 backbone, six encoder layers, six decoder layers, and eight attention heads.
- Ablation Study: When L increases or r decreases, attention-map estimation error generally decreases; subsequent experiments therefore fix r to 8 and vary L for computation control.For r greater than 6, increasing r has little effect on estimation error and FLOPs, whereas smaller r can substantially increase FLOPs.
- Ablation Study: Query clustering, key clustering, and joint query-key clustering achieve similar AP under similar FLOPs.The comparison adjusts L to equalize computational cost across the three clustering targets.
- Final Performance: ACT with L = 32 matches Faster RCNN-DC5 with fewer FLOPs, matches DETR-DC5 on APL, and exceeds DETR by 0.3% on APL.Most AP loss occurs for small and medium objects; a few epochs of MTKD substantially improve AP across object sizes.
- Inference Time and Memory: ACT’s real-environment acceleration is consistent with its theoretical FLOPs reduction, while memory cost is significantly reduced.Inference time and memory are measured on an Nvidia GeForce GTX TITAN X with batch size 1.
5 Visualisation of Adaptive Clustering
Visualization indicates that ACT’s clusters reflect both semantic content and spatial location. Deeper encoder layers use fewer prototypes relative to queries as features become more redundant.
- Cluster Semantics: Clusters correspond to semantically and spatially similar features, including groups representing a cow, sky, and field.Queries with similar semantics and locations are more likely to be grouped together.
6 Conclusion
The paper proposes ACT to reduce computation and memory costs for object detection. It also identifies training from scratch and multi-scale FPN fusion as future directions.
- ACT is proposed to reduce computation and memory costs for object detection.
- ACT does not require retraining because it is compatible with the Transformer.
- Future work will examine ACT for training from scratch and cross-scale information fusion over multi-scale FPN.