Source-linked AI summary
SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking
Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, Shengyong Chen
TL;DR
Visual tracking must handle substantial appearance variation while existing anchor-based and proposal-based designs require sensitive tuning. SiamCAR uses an end-to-end Siamese network that decomposes tracking into per-pixel classification and regression without anchors or proposals. It achieves state-of-the-art results across challenging benchmarks, including first-place GOT-10K indicators and reported gains over SiamRPN++ on GOT-10K and LaSOT.
Problem
Visual tracking remains challenging under illumination, scale, clutter, occlusion, and pose variation, while anchor-based trackers require carefully tuned hyper-parameters and fixed shapes limit deformation handling.
Method
SiamCAR jointly trains a fully convolutional Siamese feature extractor with per-pixel classification and bounding-box regression in an anchor- and proposal-free framework.
Results
SiamCAR achieves state-of-the-art results across GOT-10K, LaSOT, and other challenging benchmarks; on GOT-10K it ranks first on all indicators and improves over SiamRPN++ by 5.2%, 5.4%, and 9.0% for AO, SR0.5, and SR0.75.
Takeaways & Limitations
The framework offers a simple, neat, and generalizable fully convolutional tracker that can be modified with specific modules for further improvement.
Takeaways & Limitations
Anchor-based trackers remain established high-speed baselines, and their performance is sensitive to relative anchor hyper-parameters and empirical tuning.
Abstract
from arXiv · showhide
By decomposing the visual tracking task into two subproblems as classification for pixel category and regression for object bounding box at this pixel, we propose a novel fully convolutional Siamese network to solve visual tracking end-to-end in a per-pixel manner. The proposed framework SiamCAR consists of two simple subnetworks: one Siamese subnetwork for feature extraction and one classification-regression subnetwork for bounding box prediction. Our framework takes ResNet-50 as backbone. Different from state-of-the-art trackers like Siamese-RPN, SiamRPN++ and SPM, which are based on region proposal, the proposed framework is both proposal and anchor free. Consequently, we are able to avoid the tricky hyper-parameter tuning of anchors and reduce human intervention. The proposed framework is simple, neat and effective. Extensive experiments and comparisons with state-of-the-art trackers are conducted on many challenging benchmarks like GOT-10K, LaSOT, UAV123 and OTB-50. Without bells and whistles, our SiamCAR achieves the leading performance with a considerable real-time speed.
1. Introduction
Visual tracking remains difficult under unconstrained appearance changes, while existing Siamese approaches rely on matching and often require costly multi-scale processing or careful tuning. SiamCAR instead performs anchor-free, per-pixel classification and regression in an end-to-end Siamese framework, achieving state-of-the-art accuracy and time cost.
- Unconstrained tracking is challenged by illumination, scale, clutter, occlusion, and extreme pose variation.
- Siamese trackers formulate tracking as target matching, but single similarity maps provide limited spatial information and motivate multi-scale search processing.
- SiamCAR decomposes tracking into per-pixel classification of location labels and regression of a relative bounding box, trained jointly end to end.
- The framework uses one response map containing category and semantic information to predict object location and bounding box directly.
- SiamCAR is both anchor- and proposal-free, significantly reducing hyper-parameters and complicated training-time parameter tuning.
- Without bells and whistles, SiamCAR achieves state-of-the-art tracking performance in both accuracy and time cost.
2. Related Works
Prior Siamese trackers improved efficiency through correlation and region proposals, but anchor-based designs remain sensitive to hyper-parameter tuning and fixed shape assumptions. SiamCAR addresses these constraints with a simpler fully convolutional, anchor-free classification-regression framework.
- Siamese online-training and offline-tracking methods with deep neural networks have achieved a balance between tracking accuracy and efficiency.
- SiamFC established a fully convolutional Siamese tracker, followed by models adding correlation filters, dynamic transformations, and region proposal extraction.
- SiamRPN avoids multi-scale feature extraction by jointly training classification and regression branches for region proposals, improving efficiency.
- Anchor boxes speed region proposal tracking, but their relative hyper-parameters require careful tuning and empirical tricks for ideal performance.
- SiamCAR is illustrated as a fully convolutional architecture combining Siamese feature extraction with classification-regression bounding-box prediction.
- Fixed anchor size and aspect ratio make anchor-based trackers less suitable for objects with large shape deformation and pose variation.
3. Proposed Method
SiamCAR extracts a multi-channel Siamese response map and predicts foreground, center-ness, and bounding-box geometry at every location without anchor references. Its tracking phase ranks locations with classification and scale-change scores, then averages the top-k neighboring boxes.
- 3.1. Feature Extraction with Siamese Subnetwork: The Siamese subnetwork uses shared backbone branches for the template and search region, whose feature maps are combined by channel-by-channel correlation.The resulting response map preserves the channel dimension for downstream classification and regression.
- 3.1. Feature Extraction with Siamese Subnetwork: Multi-layer deep features aggregate low-level visual attributes and high-level semantic information for tracking.The backbone is a modified ResNet-50, and the concatenated feature representation contains 3 × 256 channels.
- 3.2. Bounding Box Prediction with Classification and Regression Subnetwork: SiamCAR directly predicts boxes at locations instead of using anchor boxes, enabling fully convolutional end-to-end training without anchor hyper-parameter tuning.A center-ness branch suppresses low-quality predictions from locations far from the object center.
- 3.2. Bounding Box Prediction with Classification and Regression Subnetwork: Each response-map location is classified as foreground or background and regressed to four distances from that location to the bounding-box sides.The regression vector is t(i, j) = (l, t, r, b), while classification outputs a two-dimensional foreground/background score.
- 3.2. Bounding Box Prediction with Classification and Regression Subnetwork: The training objective combines classification, center-ness, and regression losses, with empirically selected weights λ1 = 1 and λ2 = 3.Classification uses cross-entropy, while bounding-box regression uses IOU loss.
- 3.3. The Tracking Phase: During tracking, SiamCAR selects the highest-scoring location, then averages the top 3 boxes among 8 neighboring points for stable predictions.Neighboring points are ranked using clsij × pij, where pij is a scale-change penalty; k = 3 was found empirically to provide stable results.
4. Experiments
SiamCAR is evaluated on GOT-10K, LaSOT, OTB-50, and UAV123 using the specified training and testing protocols. It achieves leading accuracy across these benchmarks while maintaining real-time speed.
- Results on GOT-10K: 5.2%, 5.4%, and 9.0% relative gains over SiamRPN++ are reported for GOT-10K AO, SR0.5, and SR0.75, respectively.SiamCAR ranks first on all reported GOT-10K indicators, under a protocol using the same training data and unseen test classes.
- Results on LaSOT: 3.1%, 1.9%, and 1.1% relative gains over SiamRPN++ are reported for LaSOT’s three indicators.Against the provided baselines, the reported relative improvements exceed 14%, 13.7%, and 11%.
- Results on OTB-50: SiamCAR achieves the best reported results on OTB-50’s listed challenging aspects and improves accuracy especially for low resolution, out-of-plane rotation, and background clutter.The authors connect these results to semantic information implicitly decoded by the classification-regression subnetwork.
- Results on UAV123: SiamCAR outperforms the other trackers on both UAV123 indicators while using a simpler network and no heuristic tuning parameters.The comparison includes nine state-of-the-art approaches, including SiamRPN++, SiamRPN, SiamFC, and ECO.
- Run-time evaluation: 52.27 FPS is reported for SiamCAR on GOT-10K, indicating real-time evaluation speed.The speed is measured on a machine with one RTX 2080ti.
5. Conclusions
SiamCAR presents a simple Siamese classification-regression framework that solves visual tracking per pixel and achieves state-of-the-art results across challenging benchmarks.
- SiamCAR end-to-end trains a deep Siamese network for visual tracking using a fully convolutional framework.
- The framework decomposes tracking into per-pixel classification and bounding-box regression.
- SiamCAR achieves state-of-the-art results on GOT-10K and other challenging benchmarks without bells and whistles.
- Its state-of-the-art results on LaSOT demonstrate generalizability across a large dataset.
- The simple and neat framework can be modified with specific modules for further improvement.