Source-linked AI summary
Efficient Visual Tracking with Exemplar Transformers
Philippe Blatter, Menelaos Kanakis, Martin Danelljan, Luc Van Gool
TL;DR
Visual tracking models have become more accurate but often incur higher runtime costs, while efficient architectures have received limited attention. The paper introduces Exemplar Transformer, a single instance-level attention design integrated into E.T.Track, which operates in realtime on CPUs and outperforms previous realtime trackers across the reported benchmarks.
Problem
Accurate transformer-based trackers often hinder runtime, while efficient deep tracking architectures have received limited attention for computationally constrained realtime applications.
Method
The paper introduces Exemplar Transformer, a transformer layer based on single instance-level Exemplar Attention, and integrates it into a Siamese tracker.
Results
E.T.Track runs at 46.8 FPS on a CPU, achieves 59.1% AUC on LaSOT, and outperforms previous realtime trackers on five benchmarks.
Takeaways & Limitations
Exemplar Transformer provides a lightweight tracking layer that improves tracking performance with minimal or insignificant runtime impact.
Takeaways & Limitations
The design assumes that one global query value is sufficient to identify the object in single-instance tracking.
Abstract
from arXiv · showhide
The design of more complex and powerful neural network models has significantly advanced the state-of-the-art in visual object tracking. These advances can be attributed to deeper networks, or the introduction of new building blocks, such as transformers. However, in the pursuit of increased tracking performance, runtime is often hindered. Furthermore, efficient tracking architectures have received surprisingly little attention. In this paper, we introduce the Exemplar Transformer, a transformer module utilizing a single instance level attention layer for realtime visual object tracking. E.T.Track, our visual tracker that incorporates Exemplar Transformer modules, runs at 47 FPS on a CPU. This is up to 8x faster than other transformer-based models. When compared to lightweight trackers that can operate in realtime on standard CPUs, E.T.Track consistently outperforms all other methods on the LaSOT, OTB-100, NFS, TrackingNet, and VOT-ST2020 datasets. Code and models are available at https://github.com/pblatter/ettrack.
1. Introduction
The paper targets the runtime costs of increasingly accurate transformer-based visual trackers by introducing Exemplar Attention and integrating it into E.T.Track. E.T.Track is designed for realtime CPU tracking while maintaining strong benchmark performance.
- Motivation: Transformer advances improve tracking accuracy and robustness but often increase computational cost, limiting realtime operation.The paper motivates efficient tracking for applications requiring operation on computationally limited hardware.
- Method: Exemplar Attention uses a single instance-level attention layer for visual tracking.Its design assumes one global query can represent a single tracked object and uses exemplar values as shared dataset memory.
- Architecture: E.T.Track integrates the Exemplar Transformer layer into a Siamese tracking architecture, replacing convolutional layers in the tracker heads.The authors report improved performance with an insignificant runtime increase compared with the mobile LT-Mobile tracker.
- Results: 46.8 FPS on a CPU and 59.1% AUC on LaSOT establish E.T.Track as a state-of-the-art realtime CPU tracker.The evaluation covers six benchmark datasets: LaSOT, OTB-100, UAV-123, NFS, TrackingNet, and VOT-ST2020.
- Results: The paper reports realtime CPU operation while outperforming previous realtime trackers on five benchmarks.This is presented as a central contribution of the proposed tracker.
2. Related Work
Prior work includes Siamese trackers, transformer-based tracking, and efficient tracking architectures. The paper positions Exemplar Transformer as a lightweight, task-specific alternative that can improve tracking performance with negligible runtime effects.
- Siamese Trackers: Siamese trackers formulate visual tracking as template matching using cross-correlation between search and image patches.The related work traces this framework from SiamFC through region-proposal and more recent efficient trackers.
- Transformers in Tracking: Transformer-based trackers improve visual tracking by modeling relationships such as global spatio-temporal feature dependencies.Transformers have also been applied broadly to vision tasks including classification, detection, and dense prediction.
- Efficient Tracking Architectures: Efficient tracking architectures address realtime operation on computationally limited hardware, but hand-crafted-feature methods can underperform newer complex methods.The paper identifies realtime efficiency as important for applications including autonomous driving and human-computer interfaces.
- Efficient Transformers: The paper compares Exemplar Attention with standard scaled dot-product attention using matching blocks and tensor thickness to indicate correspondence and size.The comparison is presented in Figure 2.
- Efficient Tracking Architectures: Exemplar Transformer acts as a drop-in replacement for convolutional layers, increasing performance with negligible effects on runtime.This positions the module as an efficiency-oriented alternative within tracking architectures.
3. Efficient Tracking with Transformers
The section introduces Exemplar Transformers as lightweight single-instance attention modules designed to balance tracking performance and realtime runtime. E.T.Track integrates them into a Siamese tracker by replacing tracker-head convolutions while retaining a lightweight backbone.
- Exemplar Transformers: Exemplar Transformers use single instance-level attention to reduce the cost of tracking a single object.The design treats direct connections among all spatial features as potentially suboptimal for single-object tracking.
- Standard Transformer: Standard self-attention computes relationships among all input positions, with computational complexity O(N^2D).Queries, keys, and values are projected from the input sequence before attention adapts its representation.
- Exemplar Attention: Exemplar Attention compresses the feature map with adaptive average pooling, projects a global query, and uses a small set of exemplar representations.Setting S = 1 yields one global query, while the experiments use E = 4 exemplars.
- E.T.Track: Figure 3 depicts the Siamese tracking pipeline with Exemplar Transformers incorporated in the tracker head.The architecture uses the lightweight LT-Mobile backbone and applies the transformer layer within the tracker heads.
- E.T.Track: E.T.Track integrates the Exemplar Transformer into a lightweight Siamese tracker by replacing convolutions in the classification and bounding-box regression heads.The lightweight backbone remains unchanged, and the tracker processes search and template frames before cross-correlation and parallel head predictions.
4. Experiments
Experiments evaluate E.T.Track against realtime, transformer-based, and established trackers across six benchmarks, with CPU speed reported alongside tracking performance. E.T.Track achieves strong realtime results, including 46.8 FPS on CPU and leading or competitive scores across the evaluated datasets.
- LaSOT: 59.1% AUC on LaSOT exceeds DiMP by 2.2% and LT-Mobile by 7% while maintaining comparable speed.
- NFS: 59% AUC on NFS outperforms all realtime trackers by at least 3.7%.
- OTB-100: 67.8% AUC on OTB-100 makes E.T.Track the best-performing realtime tracker, although TrDiMP reaches 71.1%.
- TrackingNet: On TrackingNet, E.T.Track improves LT-Mobile by 1.05% in precision, 2.42% in normalized precision, and 2.48% in AUC.
- VOT-ST2020: On VOT-ST2020, E.T.Track improves robustness over the lightweight convolutional baseline by 5.2%.
- Ablation study: Adding the lightweight FFN produces the final E.T.Track model and raises LaSOT AUC by 5.5% over the corresponding configuration.
5. Conclusion
The conclusion presents Exemplar Transformer as a transformer layer for single-object tracking that combines instance-level attention with exemplar representations. Integrated into a Siamese tracker, it improves tracking performance while retaining realtime operation on standard CPUs.
- Exemplar Transformer uses a single instance-level attention layer and can substitute for convolutional layers throughout the architecture.
- A single query token and jointly learned exemplar representations provide the layer’s core design.
- E.T.Track improves performance over convolutional and other transformer variants with minimal impact on overall runtime.
- E.T.Track runs in realtime on computationally limited devices such as standard CPUs.
A. Ablation of Number of Query Vectors
The supplementary ablation evaluates how many query vectors to use in Exemplar Attention. The results support the experimental choice of a single global query token for single-object tracking.
- S = 1 represents a single global query token intended to encapsulate sufficient information for single-object tracking.
- The ablation computes individual query vectors from S × S patches of the input feature map.
- S = 1 yields consistently better AUC results across the NFS, OTB, and LaSOT datasets.
B. Algorithm
The supplementary material points to pseudocode for the Exemplar Attention layer, corresponding to Equation 6 and the right side of Figure 2.
- Algorithm 1 provides pseudocode for the Exemplar Attention layer defined in Equation 6.
- The algorithm is depicted on the right side of Figure 2.
C. VOT-RT2020
On VOT-RT2020, E.T.Track achieves robustness comparable to LT-Mobile in accuracy while improving robustness by nearly 6%.
- VOT-RT2020: Nearly 6% better robustness than LT-Mobile, while maintaining comparable accuracy on VOT-RT2020.The evaluation concerns bounding-box-predicting trackers on the anchor-based short-term tracking dataset.
- Exemplar Attention: The Exemplar Attention implementation computes a query, compares it with exemplar keys, and combines the resulting similarities with exemplar values.The pseudocode shows softmax similarity, scaling by √d_k, and multiplication with exemplar values before the output is applied to X.
- Robustness: Learning exemplar representations together with an image-level query representation increases robustness relative to the convolutional counterpart.
D. Video Visualizations
Sequence visualizations show E.T.Track handling partial out-of-view motion, appearance and scale changes, target disambiguation, and recovery challenges, while also exposing shared pipeline limits.
- Sequence comparisons: E.T.Track completely recovers when the UAV-123 person re-enters the frame, with LT-Mobile yielding comparable results.
- Sequence comparisons: 88% average overlap on OTB’s Human7 is 7% higher than LT-Mobile despite video jitter.
- Sequence comparisons: E.T.Track maintains the boat track after a 180-degree turn despite appearance and scale changes, unlike LT-Mobile.
- Sequence comparisons: The Exemplar Transformer layer separates a player’s head from the basketball in NFS basketball-3, unlike LT-Mobile.
- Limitations: Both trackers struggle when a target leaves and re-enters outside the search range with changed appearance, reflecting shared inference-pipeline limitations.The pipeline penalizes size and aspect-ratio changes during box refinement and searches only a small patch near the previous location.
E. Attributes
Across LaSOT attributes, E.T.Track consistently outperforms realtime trackers, while remaining below the non-realtime STARK baseline by 7.3% on average.
- Attribute results: 10.4% is the largest gain over LT-Mobile for Full Occlusion, followed by 9.7% for Motion Blur.
- Attribute results: 8.5% and 7.5% gains over LT-Mobile occur for Background Clutter and Fast Motion, respectively.
- Attribute results: E.T.Track consistently outperforms other realtime trackers across every evaluated LaSOT attribute.
- Interpretation: The authors connect robustness gains to Exemplar Transformer layers, including improvements on attributes known as framework limitations.
- Comparison with non-realtime tracking: −7.3% average performance separates E.T.Track from non-realtime STARK, with Viewpoint Change, Full Occlusion, Fast Motion, Out-of-View, and Low Resolution most challenging.
F. Additional Success Plots
Additional success plots compare realtime and non-realtime trackers across NFS, OTB-100, and UAV-123, showing E.T.Track generally ahead of LT-Mobile and narrowing the gap with complex transformer trackers.
- Evaluation setup: The plotted datasets are NFS, OTB-100, and UAV-123, with efficient-tracker comparisons limited to LightTrack’s mobile architecture.
- Success plots: E.T.Track outperforms LT-Mobile on all but one of the benchmark datasets shown in the success plots.
- Success plots: The results highlight a shrinking gap between complex transformer trackers and E.T.Track while retaining realtime CPU operation.
- Attribute table: Table S.9 encodes dataset attributes by columns and distinguishes non-realtime trackers with grey highlighting.
- Plot encoding: The success plots compare CPU-realtime trackers with continuous warm-colour lines against non-realtime trackers with dashed cool-colour lines.