Source-linked AI summary
SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines
Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, Gang Yu
TL;DR
Generic visual tracking requires robust classification and accurate target-state estimation without object-class or environmental prior knowledge. SiamFC++ introduces four tracking-specific design guidelines and implements them with anchor-free classification, regression, and quality assessment. It reaches state-of-the-art performance across five benchmarks, including TrackingNet AUC 75.4 at over 90 FPS.
Problem
Existing trackers use diverse target-state estimation methods, but few explicitly account for the particular requirements of generic visual tracking.
Method
SiamFC++ applies four guidelines through separate classification and state-estimation branches, non-ambiguous anchor-free scoring, prior-knowledge-free tracking, and estimation-quality assessment.
Results
SiamFC++ achieves state-of-the-art performance on five challenging benchmarks, including TrackingNet AUC 75.4 at over 90 FPS.
Takeaways & Limitations
The proposed guidelines and tracker support robust, accurate, fast, and generalized generic object tracking.
Takeaways & Limitations
Anchor-based feature matching uses fixed-aspect-ratio patches while assigning anchors with varying aspect ratios, contributing to ambiguity in SiamRPN++.
Abstract
from arXiv · showhide
Visual tracking problem demands to efficiently perform robust classification and accurate target state estimation over a given target at the same time. Former methods have proposed various ways of target state estimation, yet few of them took the particularity of the visual tracking problem itself into consideration. After a careful analysis, we propose a set of practical guidelines of target state estimation for high-performance generic object tracker design. Following these guidelines, we design our Fully Convolutional Siamese tracker++ (SiamFC++) by introducing both classification and target state estimation branch(G1), classification score without ambiguity(G2), tracking without prior knowledge(G3), and estimation quality score(G4). Extensive analysis and ablation studies demonstrate the effectiveness of our proposed guidelines. Without bells and whistles, our SiamFC++ tracker achieves state-of-the-art performance on five challenging benchmarks(OTB2015, VOT2018, LaSOT, GOT-10k, TrackingNet), which proves both the tracking and generalization ability of the tracker. Particularly, on the large-scale TrackingNet dataset, SiamFC++ achieves a previously unseen AUC score of 75.4 while running at over 90 FPS, which is far above the real-time requirement. Code and models are available at: https://github.com/MegviiDetection/video_analyst .
1 Introduction
Generic visual tracking must locate an unknown moving object while jointly achieving robust classification and accurate target-state estimation. SiamFC++ addresses shortcomings in prior estimation strategies with four design guidelines and reports state-of-the-art benchmark performance.
- Generic tracking locates a moving object from limited information, without prior knowledge of the object class or surrounding environment.
- Prior trackers differ substantially in target-state estimation, including multi-scale testing, iterative refinement, and anchor-based proposal methods.
- The proposed guidelines require separate classification and state-estimation tasks, non-ambiguous scoring, no prior knowledge, and estimation-quality assessment.
- SiamFC++ implements these guidelines with parallel classification and regression branches, anchor-free scoring, and a quality-assessment branch.
- SiamFC++ achieves state-of-the-art results on five challenging benchmarks, including an AUC score of 75.4 on TrackingNet at over 90 FPS.
2 Related Works
Prior visual trackers estimate target state through multi-scale testing, iterative refinement, or anchor-based proposals, each with important accuracy, efficiency, or ambiguity concerns. SiamFC++ is motivated by a tracker design that better matches generic tracking requirements.
- Modern trackers use three main target-state estimation strategies: multi-scale testing, sequential classification and refinement, or Siamese RPN proposals.
- Multi-scale testing selects the scale with the highest classification score, but bounding-box estimation requires higher-level understanding of object pose.
- ATOM iteratively refines boxes from classification-based locations, improving accuracy while incurring heavy computation and additional hyperparameters.
- SiamRPN methods use predefined anchors to regress shifts and size differences, although this proposal structure is designed for detection and can create ambiguity in single-object tracking.
- The SiamFC++ framework is presented as a fully convolutional Siamese pipeline with feature extraction and classification, regression, and quality-related operations.
3 SiamFC++: Fully Convolutional Siamese Tracker for Object Tracking
SiamFC++ uses Siamese feature matching with separate classification and regression heads, treating feature-map locations directly rather than relying on predefined anchors. A quality branch combines estimated quality with classification scores to select more accurate boxes.
- Framework: SiamFC++ uses a Siamese subnetwork for feature extraction and a proposal subnetwork for classification and regression.
- Siamese-based Feature Extraction and Matching: The template branch processes the first-frame target and the search branch processes the current frame, with cross-correlation performed in embedding space.
- Head Network: Separate classification and regression heads classify each corresponding image patch and predict offsets for refining bounding-box locations.
- Non-ambiguous Scoring: Each feature-map location corresponds to an input-image patch, and SiamFC++ directly treats locations as training samples instead of assigning multiple anchor boxes.
- Prior Knowledge-free Tracking: SiamFC++ is free of predefined anchor boxes and associated scale/ratio prior knowledge because classification and regression are performed with respect to locations.
- Quality Assessment: The quality branch predicts Prior Spatial Score, while an IoU score is an alternative quality target; inference multiplies quality and classification scores to select the final box.
- Training Objective: The training objective combines focal classification loss, binary-cross-entropy quality loss, and IoU regression loss.
4 Experiments
Experiments evaluate SiamFC++ across implementation settings, ablations, benchmarks, and comparisons with trackers that do not follow the proposed guidelines. Results show strong accuracy, robustness, generalization, and speed, while analyses identify the regression branch and guideline violations as important factors.
- Implementation and speed: 160 FPS with AlexNet and about 90 FPS with GoogLeNet are reported on the VOT2018 short-term benchmark.Both measurements use an NVIDIA RTX 2080Ti GPU.
- Ablation study: The regression branch contributes the largest listed improvement, with a ΔEAO of 0.094, followed by data source diversity, stronger backbone, and better head structure.The corresponding listed improvements are 0.063/0.010, 0.026, and 0.020.
- Benchmark results: SiamFC++ reaches a success score of 0.682 on OTB2015, described as state-of-the-art among the compared trackers.OTB2015 contains 100 evaluation videos.
- Benchmark results: On VOT2018, SiamFC++ achieves an EAO of 0.400 while running over 100 FPS and has a reported robustness advantage among compared trackers.The AlexNet version outperforms trackers with the same speed, while the GoogLeNet version yields a comparable score.
- Benchmark results: On GOT-10k, the AlexNet and GoogLeNet versions achieve AO scores of 53.5 and 59.5, respectively, using only the training subset.The GoogLeNet version is reported as superior to ATOM, and the result is presented as evidence of generalization to unseen target classes.
- Benchmark results: SiamFC++-GoogLeNet outperforms compared methods on TrackingNet precision and success rate, while the lightweight version balances performance and speed without Youtube-BB training data.The evaluation uses 511 videos from the TrackingNet test split.
- Guideline analysis: SiamRPN++ shows similar score distributions for successful and failed tracking, whereas SiamFC++ exhibits different failure-state and success-state score patterns.The comparison is used to analyze classification-score ambiguity under challenging tracking conditions.
- Guideline analysis: SiamRPN++ performance varies with anchor scale and ratio settings, and its predicted boxes tend to overlap anchors more than ground-truth boxes.The passages associate these effects with prior-knowledge dependence and bias in target state estimation.
5 Conclusion
The paper proposes guidelines for target state estimation and instantiates them in SiamFC++, achieving state-of-the-art performance across five challenging benchmarks while running at 90 FPS.
- SiamFC++ combines classification and target state estimation, ambiguity-free classification scores, tracking without prior knowledge, and estimation quality assessment.
- Extensive ablation studies verify the effectiveness of the proposed target-estimation guidelines.
- SiamFC++ achieves state-of-the-art performance on five challenging benchmarks while running at 90 FPS.
A Backbone Choice
Using a GoogLeNet backbone, SiamFC++ matches or exceeds the performance of previous ResNet-50 trackers across benchmarks while using fewer computational resources.
- The GoogLeNet version achieves the same or better performance than previous ResNet-50 trackers across different benchmarks.
- Smaller MACs and parameter amounts contribute significantly to the tracker's efficiency.
B Test Phase Behavior
During testing, SiamFC++ produces dense bounding-box and confidence-score outputs, then applies penalties, a cosine window, score-based selection, and smooth size updating to obtain robust tracking results.
- The model outputs N × N × 4 bounding boxes and N × N confidence scores for the search image.
- Confidence scores are penalized according to box ratio and size changes before further post-processing.
- A cosine window suppresses large displacement under the assumption of smooth motion.
- The highest-scoring point after post-processing selects the current bounding box estimate.
- The target size is updated by linear interpolation, with the update speed controlled by a hyperparameter and penalized confidence.
C Ablation Study over SiamFC++
Ablations show that data diversity, backbone capacity, and head structure affect SiamFC++ performance, while two head convolution layers provide the selected speed–accuracy balance.
- Data source diversity: Increasing training-data diversity significantly improves tracker performance.
- Backbone capacity: The GoogLeNet backbone outperforms AlexNet by 0.017 on VOT2018 and 3.2 on GOT-10k.
- Structure of head: Qualitative results show robustness to fast motion, pose change, occlusion, and distractors.
- Structure of head: Adding the second and third conv3 × 3 layers improves performance by 0.4 and 0.3, respectively, despite shrinking the response region.
- Structure of head: Two conv3 × 3 layers are selected as a balance between performance and speed.
D Data Distribution of Different Datasets
Dataset scale and aspect-ratio distributions differ, and these distributions can function as prior knowledge. Generic trackers should therefore avoid relying on scale/ratio-related factors.
- Scale and aspect-ratio distributions vary across datasets.The paper collects scale/ratio statistics to characterize this variation.
- Dataset-specific distributions can constitute a form of prior knowledge.The passage links distributional statistics to prior information available during tracker design.
- Generic object trackers should prevent involvement of scale/ratio-related factors.The guideline follows from treating access to dataset-specific prior knowledge as contrary to generic tracking.
E Comment over Anchor-based Scoring: A Maxout Perspective
Anchor-based tracking scores can be viewed as foreground-score maxout at each pixel. In single-object tracking, false positives are more damaging than false negatives because they can cause drift, motivating ambiguity avoidance in SiamFC++.
- Anchor-based tracking performs foreground-score maxout at each pixel location.This perspective applies under the described post-processing scheme.
- Foreground maxout favors recall and reduces false negatives, whereas background maxout suppresses false positives.These tendencies are identified from face-detection practice.
- False positives are more harmful than false negatives in single-object tracking because they can cause drift.False negatives instead produce low scores and preserve the previous bounding box under cosine-window post-processing.
- Avoiding foreground-score maxout through per-pixel scoring contributes to SiamFC++ robustness.The paper presents ambiguity avoidance as the mechanism for removing this maxout behavior.