Source-linked AI summary
A Twofold Siamese Network for Real-Time Object Tracking
Anfeng He, Chong Luo, Xinmei Tian, Wenjun Zeng
TL;DR
Real-time object tracking needs features that are both discriminative and tolerant of target appearance changes without costly online training. SA-Siam combines separately trained semantic and appearance Siamese branches with semantic channel attention, and it outperforms other real-time trackers by a large margin on OTB benchmarks.
Problem
Real-time tracking must localize an unknown target despite clutter, motion, deformation, appearance changes, and the computational cost of online CNN training.
Method
SA-Siam uses separately trained semantic and appearance Siamese branches, combining their similarity scores at testing time and applying channel attention to semantic features.
Results
SA-Siam outperforms all other real-time trackers by a large margin on OTB-2013/50/100 benchmarks.
Takeaways & Limitations
Keeping semantic and appearance branches heterogeneous lets SA-Siam benefit from their complementarity without early fusion.
Takeaways & Limitations
The semantic network is not fine-tuned, and the appearance branch does not use multilevel features or channel attention because the authors expect limited gain relative to added cost.
Abstract
from arXiv · showhide
Observing that Semantic features learned in an image classification task and Appearance features learned in a similarity matching task complement each other, we build a twofold Siamese network, named SA-Siam, for real-time object tracking. SA-Siam is composed of a semantic branch and an appearance branch. Each branch is a similarity-learning Siamese network. An important design choice in SA-Siam is to separately train the two branches to keep the heterogeneity of the two types of features. In addition, we propose a channel attention mechanism for the semantic branch. Channel-wise weights are computed according to the channel activations around the target position. While the inherited architecture from SiamFC \cite{SiamFC} allows our tracker to operate beyond real-time, the twofold design and the attention mechanism significantly improve the tracking performance. The proposed SA-Siam outperforms all other real-time trackers by a large margin on OTB-2013/50/100 benchmarks.
1. Introduction
Real-time visual tracking must localize an initially unknown target despite motion, deformation, appearance changes, and cluttered backgrounds. SA-Siam addresses SiamFC’s limited generalization by combining separately trained semantic and appearance branches with semantic channel attention.
- Real-time tracking must distinguish an unknown target from clutter while tolerating motion, deformation, and appearance changes.
- Online CNN training can improve tracking but is computationally expensive, leaving most online CNN-based trackers slower than real-time.
- SiamFC avoids online training through similarity learning and fully convolutional processing, but its generalization capability remains poor.
- SA-Siam combines semantic features from image classification with appearance features from similarity learning in two separately trained Siamese branches.
- A channel attention mechanism weights semantic feature channels using target and surrounding-context responses for target adaptation.
- SA-Siam outperforms other real-time trackers by a large margin on OTB-2013/50/100 benchmarks and achieves state-of-the-art performance on VOT benchmarks.
2. Related Work
Siamese and ensemble trackers use similarity learning or heterogeneous CNN features to achieve efficient tracking. SA-Siam extends SiamFC with separately trained appearance and semantic branches plus channel-wise attention for target adaptation.
- 2.1. Siamese Network Based Trackers: Similarity learning compares a target patch with candidate patches and can avoid online training, supporting real-time tracking.
- 2.1. Siamese Network Based Trackers: SiamFC evaluates similarity across translated sub-windows on a dense grid using a fully convolutional network.
- 2.1. Siamese Network Based Trackers: Follow-up trackers modify SiamFC with early stopping, correlation filters, optical flow, or online embedding updates, with SINT operating at 4 frames per second.
- 2.1. Siamese Network Based Trackers: SA-Siam inherits SiamFC’s architecture while introducing heterogeneous feature utilization through separately trained branches.
- 2.2. Ensemble Trackers: Ensemble trackers commonly combine CNN features from different layers while reducing correlation across models.
- 2.2. Ensemble Trackers: SA-Siam applies channel-wise attention to channel activations as a form of target adaptation that potentially improves tracking performance.
3. Our Approach
SA-Siam combines separately trained appearance and semantic Siamese branches for real-time tracking, using channel attention to adapt semantic features to the target. The branches produce response maps that are fused during testing to localize the target.
- Twofold network: SA-Siam uses appearance features from similarity learning and semantic features from image classification in a twofold fully convolutional Siamese network.The design treats the two feature types as complementary for robust visual tracking.
- Fusion and training: The two branches are trained separately and combined only at testing, when their heat maps are averaged with parameter λ and the maximum response indicates the target center.Three input scales are used to address scale changes while balancing performance and speed.
- Appearance branch: The appearance branch clones SiamFC and trains its feature extractor from scratch with training pairs and ground-truth response maps using logistic loss.Its response map is computed by correlating target and search-region features.
- Semantic branch: The semantic branch uses a fixed image-classification CNN, extracts conv4 and conv5 features, and fuses them with a 1 × 1 ConvNet for correlation.The target input includes surrounding context, while low-level features are not extracted.
- Design rationale: High-level semantic features improve generalization to appearance changes but are less discriminative, motivating channel attention to strengthen the semantic branch.The method adapts channel importance using the target and its surrounding context.
- Channel attention: Channel attention computes target-adaptive weights from pooled feature-map grids using a two-layer MLP and a biased Sigmoid, then weights semantic target activations.The center grid corresponds to the target, and the attention module runs only once during the first frame.
4. Experiments
Experiments evaluate SA-Siam’s implementation, ablations, attention behavior, and performance across OTB and VOT benchmarks. The results support separately trained complementary branches, channel attention, and real-time operation.
- Ablation analysis: The combined semantic and appearance model achieves a large improvement over either branch alone and reaches state-of-the-art performance without multilevel features or channel attention.The ablation compares semantic-only S1, appearance-only A1, and their combined SA-Siam model.
- Ablation analysis: Ensembling appearance models A1A2 or semantic models S1S2 performs worse than SA-Siam, supporting complementary features across the two branches.The comparison is conducted on OTB-2013.
- Ablation analysis: Channel attention provides significant gains by balancing intra-layer and inter-layer channel importance, whereas directly adding multilevel features is only slightly helpful.The attention module is evaluated as part of the last three rows of Table 1.
- Ablation analysis: Attention assigns different channel-weight distributions to conv4 and conv5 and produces video-dependent conv4 weights.For video bolt, the module tends to suppress more conv4 channels; weights lie in the range [0.5, 1.5].
- Ablation analysis: Jointly training the two branches underperforms separate training, with OTB AUC and precision pairs of (0.630, 0.831), (0.546, 0.739), and (0.620, 0.819).The pairs correspond respectively to OTB-2013, OTB-50, and OTB-100.
- Implementation and evaluation: SA-Siam runs at 50 fps, exceeding the paper’s 25 fps real-time threshold.The model is implemented in TensorFlow on a Xeon E5 CPU with a GeForce GTX Titan X GPU.
- OTB benchmarks: SA-Siam achieves the best performance among compared real-time trackers on OTB-2013, OTB-50, and OTB-100.Precision and success plots are generated with the OTB toolkit.
- VOT benchmarks: On VOT2015, SA-Siam has the highest accuracy among the compared real-time trackers; on VOT2016 it is the most robust real-time tracker, with accuracy and EAO among the top three.On VOT2017, SA-Siam remains among the top tier across all evaluation metrics.
5. Conclusion
The paper presents SA-Siam, a twofold Siamese network that combines heterogeneous semantic and appearance features without early fusion, alongside channel attention for target adaptation. It reports strong OTB and VOT performance relative to real-time trackers.
- Conclusion: SA-Siam uses separately trained semantic and appearance branches to preserve the heterogeneity of their features.The branches are combined only at testing time rather than fused at an early stage.
- Conclusion: A channel attention module provides target adaptation in the semantic branch.The module complements the two-branch design by weighting channels according to target-related activations.
- Conclusion: SA-Siam outperforms other real-time trackers by a large margin on OTB benchmarks and performs favorably across the VOT benchmark series.The conclusion summarizes the reported benchmark results.