Source-linked AI summary
STMTrack: Template-free Visual Tracking with Space-time Memory Networks
Zhihong Fu, Qingjie Liu, Zehua Fu, Yunhong Wang
TL;DR
Fixed first-frame templates struggle with target appearance changes, while template updating can be computationally costly and strategically complex. STMTrack uses a space-time memory network to retrieve historical target information for current-frame localization, and it outperforms prior state-of-the-art real-time methods at 37 FPS. Its VOT2018 accuracy is constrained by using axis-aligned rather than rotated bounding boxes.
Problem
Fixed first-frame templates are poorly capable of resisting target appearance changes, while template updating requires time-consuming optimization and complex hand-designed strategies that hinder real-time tracking.
Method
STMTrack uses multiple historical frames and foreground-background label maps in a space-time memory network, retrieving target information for the query frame and combining it with query features.
Results
STMTrack surpasses all state-of-the-art real-time approaches across six benchmarks while running at 37 FPS.
Takeaways & Limitations
Historical information and current-frame-determined retrieval give STMTrack strong adaptive ability to target variations without traditional template updating.
Takeaways & Limitations
On VOT2018, accuracy is worse than D3S and SiamAttn because the tracker estimates axis-aligned boxes while evaluation ground truths are rotated boxes.
Abstract
from arXiv · showhide
Boosting performance of the offline trained siamese trackers is getting harder nowadays since the fixed information of the template cropped from the first frame has been almost thoroughly mined, but they are poorly capable of resisting target appearance changes. Existing trackers with template updating mechanisms rely on time-consuming numerical optimization and complex hand-designed strategies to achieve competitive performance, hindering them from real-time tracking and practical applications. In this paper, we propose a novel tracking framework built on top of a space-time memory network that is competent to make full use of historical information related to the target for better adapting to appearance variations during tracking. Specifically, a novel memory mechanism is introduced, which stores the historical information of the target to guide the tracker to focus on the most informative regions in the current frame. Furthermore, the pixel-level similarity computation of the memory network enables our tracker to generate much more accurate bounding boxes of the target. Extensive experiments and comparisons with many competitive trackers on challenging large-scale benchmarks, OTB-2015, TrackingNet, GOT-10k, LaSOT, UAV123, and VOT2018, show that, without bells and whistles, our tracker outperforms all previous state-of-the-art real-time methods while running at 37 FPS. The code is available at https://github.com/fzh0917/STMTrack.
1. Introduction
Visual tracking remains difficult under occlusions, fast motion, and non-rigid deformation, while fixed templates adapt poorly and template updating can undermine real-time use. STMTrack replaces template updating with historical memory and reports stronger real-time benchmark performance.
- Visual tracking supports human-computer interaction, video surveillance, and autonomous driving but remains challenged by occlusions, fast motions, and non-rigid deformations.
- Offline siamese trackers rely on a template from the first frame, whereas template-updating trackers require substantial computation and carefully tuned strategies.
- STMTrack uses historical target information in a memory network instead of a template and its online updating, while using pixel-level similarity for localization.
- 37 FPS: STMTrack surpasses state-of-the-art real-time approaches across six benchmarks, including OTB-2015, TrackingNet, LaSOT, GOT-10k, UAV123, and VOT2018.On TrackingNet, it achieves 80.3 success (AUC), outperforming the previous best real-time method by 4.5%.
2. Related Work
Prior tracking methods commonly match a first-frame template or update it through specialized mechanisms, while memory networks offer another way to use historical information. STMTrack retrieves historical target information according to the current frame rather than relying on a learned controller or template synthesis.
- Siamese trackers match a first-frame template to current search regions and provide real-time performance in routine scenarios.Their vulnerability appears under drastic appearance changes, non-rigid deformations, and partial occlusions.
- Template updating improves adaptation to target variation but can require computational resources and customized strategies.
- Memory networks read and write historical information through external memory components and have improved performance in few-shot learning and video object segmentation.
- MemTrack combines a residual template read by an LSTM-controlled memory network with the initial template, but its performance depends strongly on LSTM controller learning quality.
- STMTrack determines historical-information retrieval from the current frame itself, enabling adaptive use of multiple-frame information without the prior controller mechanism.
3. Proposed Method
STMTrack combines separate memory and query feature extraction with a space-time memory reader and an anchor-free head. Historical target features are retrieved through pixel-level similarities, fused with the query features, and used for classification, centerness estimation, and bounding-box regression.
- Architecture: The framework contains feature extraction, a space-time memory network, and a head network that processes historical memory frames and the current query frame.The memory branch also receives foreground-background label maps, while the query branch receives only the current frame.
- Feature Extraction: The memory branch embeds labeled memory frames and reduces their feature dimensionality to 512, while the query branch similarly reduces the query features to 512.The memory labels help the memory backbone learn target characteristics rather than distractors and cluttered background information.
- Space-time Memory Network: The reader computes a similarity matrix between every memory-feature pixel and every query-feature pixel, then normalizes the similarities with a softmax function.The similarity matrix has shape RT HW × HW, and its entries use vector dot-products with a scaling factor s.
- Space-time Memory Network: Soft similarity weights retrieve target information from historical memory features, which is concatenated with the query feature map to form the synthetic feature map y.The reader adapts the retrieved information to the query frame and produces readout information with the same spatial size as the query feature map.
- Head Network: The anchor-free head uses classification, centerness, and regression branches to classify targets, suppress off-center responses, and estimate bounding boxes.During inference, the classification and centerness response maps are multiplied before bounding-box prediction.
- Inference Phase: During inference, six memory frames are selected from historical frames, while the memory size remains independent of the number of frames used during training.The first N frames are initially used as memory frames, and subsequent sampling selects representative historical frames.
4. Experiments
Experiments evaluate STMTrack's implementation, ablations, and comparisons across six tracking benchmarks. The results show strong performance across datasets, with design choices for backbones, label maps, and memory size affecting outcomes.
- Ablation Study: Different backbones outperform a shared backbone across the ablation comparisons, although sharing improves AO by 2.3% without foreground-background label maps.The comparison is conducted on GOT-10k and other benchmarks.
- Ablation Study: Foreground-background label maps improve performance by 2.9% with the same backbone and 7.4% with different backbones.The ablation identifies these maps as crucial to the memory mechanism.
- Ablation Study: Three reference frames provide the best GOT-10k training performance in terms of average overlap (AO).Increasing the training memory size exposes more target patterns but can also produce highly similar frame pairs.
- Ablation Study: Six reference frames are most suitable during TrackingNet inference, while larger memory sizes do not consistently improve performance.The authors associate this behavior with overfitting and low-quality memory frames affecting tracking results.
- Comparison with the state-of-the-art: STMTrack surpasses previous best performance on OTB-2015 by 0.4% in success (AUC) and outperforms all previous state-of-the-art real-time approaches on TrackingNet.On TrackingNet, the test set contains 511 videos and ground truths are evaluated through the dedicated server.
5. Conclusions
STMTrack uses space-time memory networks and multiple memory frames to adaptively retrieve target information for tracking. Experiments show stronger performance than state-of-the-art real-time methods at 37 FPS.
- The framework abandons traditional template-based tracking by using multiple memory frames and foreground-background label maps to locate targets.
- Adaptive retrieval of stored target information helps the tracker handle target variations.
- The tracker achieves better performance than current state-of-the-art real-time methods while running at 37 FPS.
- Experiments indicate the framework has generalizability, extendibility, and applicability.