Source-linked AI summary
Accelerating DETR Convergence via Semantic-Aligned Matching
Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, Shijian Lu
TL;DR
DETR’s queries initially match image locations broadly, making it difficult for them to focus on relevant regions and contributing to slow convergence. SAM-DETR aligns queries with encoded image features and searches salient points for matching. It improves standalone performance over DETR and, combined with SMCA, reaches performance on par with Faster R-CNN within 12 epochs.
Problem
DETR’s object queries are initially matched broadly across encoded image locations, making it difficult for them to focus on specific regions and causing slow convergence.
Source: §3.1 · p. 3Method
SAM-DETR semantically aligns object queries with encoded image features and searches salient point features for semantic-aligned matching.
Source: §1 · p. 2Results
+10.8% AP for standalone SAM-DETR over DETR; combining SAM-DETR with SMCA improves AP by +2.9% over standalone SAM-DETR and +4.4% over SMCA-DETR, reaching parity with Faster R-CNN within 12 epochs.
Source: §4.2 · pp. 7–8Takeaways & Limitations
SAM-DETR’s plug-and-play module leaves most other operations unchanged and can be integrated with existing convergence solutions to boost performance further.
Source: §1 · p. 2, §5 · p. 8Compared with Faster R-CNN, SAM-DETR inherits DETR’s degraded performance on small objects; the authors identify multi-scale features as a future direction for improving it.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
The recently developed DEtection TRansformer (DETR) establishes a new object detection paradigm by eliminating a series of hand-crafted components. However, DETR suffers from extremely slow convergence, which increases the training cost significantly. We observe that the slow convergence is largely attributed to the complication in matching object queries with target features in different feature embedding spaces. This paper presents SAM-DETR, a Semantic-Aligned-Matching DETR that greatly accelerates DETR's convergence without sacrificing its accuracy. SAM-DETR addresses the convergence issue from two perspectives. First, it projects object queries into the same embedding space as encoded image features, where the matching can be accomplished efficiently with aligned semantics. Second, it explicitly searches salient points with the most discriminative features for semantic-aligned matching, which further speeds up the convergence and boosts detection accuracy as well. Being like a plug and play, SAM-DETR complements existing convergence solutions well yet only introduces slight computational overhead. Extensive experiments show that the proposed SAM-DETR achieves superior convergence as well as competitive detection accuracy. The implementation codes are available at https://github.com/ZhangGongjie/SAM-DETR.
1. Introduction
DETR’s slow convergence is linked to difficulty matching object queries with relevant image features, increasing training cost. SAM-DETR aligns their semantics and searches salient points to accelerate convergence while remaining complementary to existing solutions.
1. Introduction
- DETR requires 500 epochs to converge on COCO, compared with 12–36 epochs for Faster R-CNN, because matching queries to target features is difficult.The paper identifies this slow convergence as a substantial increase in training cost.Source: §1 · pp. 1–2
- SAM-DETR adds a plug-and-play module before cross-attention to project object queries and encoded image features into a semantically aligned embedding space.The aligned semantics provide a prior for queries to focus on similar image regions.Source: §1 · p. 2
- SAM-DETR explicitly searches salient points with discriminative features for semantic-aligned matching, fitting the original multi-head attention mechanism.The approach leaves most other operations unchanged, supporting integration with existing convergence solutions.Source: §1 · p. 2
- SAM-DETR reaches convergence speed comparable to Faster R-CNN within 12 training epochs and can complement existing convergence-boosting solutions.The convergence curves compare methods on COCO val 2017 under a 12-epoch, single-scale training scheme.Source: Figure 1 · p. 1, §1 · p. 2
2. Related Work
Prior object detectors often retain hand-crafted components, while DETR offers an end-to-end alternative but converges slowly. Earlier convergence methods modify attention; this work instead draws on shared-embedding matching architectures.
2. Related Work
- Many two-stage and single-stage detectors rely on hand-crafted components such as anchors, rule-based target assignment, and NMS, whereas DETR establishes a fully end-to-end paradigm.DETR uses a Transformer encoder-decoder and set-based global loss to replace those components.Source: §2 · p. 2
- Deformable DETR, Conditional DETR, and SMCA-DETR address DETR’s slow convergence by replacing or conditioning its attention, while SAM-DETR leaves the attention mechanism unchanged.The paper describes its approach as complementary to methods that modify attention.Source: §2 · p. 3
- Siamese-based architectures project both matching sides into the same embedding space, motivating semantic alignment of DETR’s object queries and encoded image features.The paper connects this matching approach to vision tasks including tracking, re-identification, and few-shot recognition.Source: §2 · p. 3
3. Proposed Method
SAM-DETR adds a Semantics Aligner to DETR’s decoder to ease matching between object queries and encoded image features. The method aligns their semantics, searches salient points, and reweights the resulting queries using previous query embeddings.
3.1. A Review of DETR
- In DETR, cross-attention matches object queries to image regions and distills relevant features, but initially queries match all locations, making it difficult to focus on relevant regions.The paper identifies this matching difficulty as a key reason for slow convergence.Source: §3.1 · p. 3
3.2. SAM-DETR
- SAM-DETR aligns object queries with encoded image features in a shared embedding space to ease matching and accelerate DETR’s convergence.The Semantics Aligner uses learnable reference boxes to guide the generation of new query embeddings before cross-attention.Source: §3.2 · p. 3
3.2.1 Semantic-Aligned Matching
- Resampling features from encoded image regions without projection puts new query embeddings in the same embedding space as those features.RoIAlign extracts region features using reference boxes, which provide a prior for queries to focus on semantically similar regions.Source: §3.2.1 · p. 4
3.2.2 Matching with Salient Point Features
- A ConvNet and MLP predict salient-point coordinates within each reference box; sampled point features are concatenated so each attention head can focus on one point.Sinusoidal position embeddings for the salient points are also concatenated for cross-attention.Source: §3.2.2 · p. 5
3.2.3 Reweighting by Previous Query Embeddings
- Previous query embeddings generate sigmoid-based reweighting coefficients that highlight useful features in new queries and their position embeddings.This lets the Semantics Aligner use information from earlier query embeddings in cross-attention.Source: §3.2.3 · p. 5
3.3. Compatibility with SMCA-DETR
- SAM-DETR adds a plug-and-play module with slight computational overhead and can complement existing convergence solutions, including SMCA-DETR.For the SMCA-DETR integration, salient-point coordinates set Gaussian-like weight-map centers, while pooled RoI features predict their scales.Source: §3.3 · p. 5
3.4. Visualization and Analysis
- After 12 epochs with ResNet-50, SAM-DETR’s salient points mostly fall on target objects and its attention maps are more precise than DETR’s scattered maps.The visual analysis reports that this narrower attention search facilitates convergence and helps attend to object extremities.Source: §3.4 · pp. 5–6, Figure 4 · p. 6
4. Experiments
On COCO, SAM-DETR improves DETR’s performance under short training and can be combined with existing convergence methods. Ablations support semantic-aligned matching and salient-point search, while the reported comparison with Faster R-CNN identifies weaker small-object performance.
4.1. Experiment Setup
- Experiments use COCO 2017, with both 12-epoch and 50-epoch training schemes.The 12-epoch schedule decays learning rate after 10 epochs; the 50-epoch schedule decays it after 40.Source: §4.1 · pp. 6–7
4.2. Experiment Results
- +10.8% AP over DETR; combining SAM-DETR with SMCA adds +2.9% AP over standalone SAM-DETR and +4.4% AP over SMCA-DETR, reaching performance on par with Faster R-CNN within 12 epochs.Across a stronger R50-DC5 backbone and 50-epoch training, SAM-DETR consistently improves DETR and reaches state-of-the-art accuracy when integrated with SMCA.Source: §4.2 · pp. 7–8, Table 1 · p. 7
4.3. Ablation Study
- SAM with query resampling outperforms the baseline; max-pooling improves on average-pooling, and salient-point resampling, especially multiple points, boosts performance.Even naive max-pooling yields a 10.7% AP0.5 improvement; the authors report that searching multiple salient points works with multi-head attention.Source: §4.3 · p. 8
- Searching within reference boxes outperforms image-scale search, and reweighting new queries with previous query embeddings consistently boosts performance.The authors suspect that the larger image-scale search space makes matching more difficult.Source: §4.3 · p. 8
4.4. Limitation
- Compared with Faster R-CNN, SAM-DETR inherits DETR’s superior accuracy on large objects but has degraded performance on small objects.The authors identify multi-scale features as a possible avenue to improve small-object accuracy.Source: §4.4 · p. 8
5. Conclusion
SAM-DETR uses a plug-and-play module to align object queries with encoded image features and searches salient-point features for matching. It integrates with existing convergence solutions and achieves accuracy comparable to Faster R-CNN within 12 training epochs.
5. Conclusion
- SAM-DETR combines semantic alignment and salient-point search in a plug-and-play module that can be integrated with existing convergence solutions to boost performance.The method facilitates matching between object queries and encoded image features; combined with existing solutions, it reaches accuracy comparable to Faster R-CNN within 12 training epochs.Source: §5 · p. 8