SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking

Dongyan Guo; Jun Wang; Ying Cui; Zhenhua Wang; Shengyong Chen

Source-linked AI summary

SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking

Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, Shengyong Chen

arXiv:1911.07241v2Nov 17, 2019cs.CV

TL;DR

Visual tracking must handle substantial appearance variation while existing anchor-based and proposal-based designs require sensitive tuning. SiamCAR uses an end-to-end Siamese network that decomposes tracking into per-pixel classification and regression without anchors or proposals. It achieves state-of-the-art results across challenging benchmarks, including first-place GOT-10K indicators and reported gains over SiamRPN++ on GOT-10K and LaSOT.

Problem
Visual tracking remains challenging under illumination, scale, clutter, occlusion, and pose variation, while anchor-based trackers require carefully tuned hyper-parameters and fixed shapes limit deformation handling.
Source: §1 · p. 1, §2 · p. 3
Method
SiamCAR jointly trains a fully convolutional Siamese feature extractor with per-pixel classification and bounding-box regression in an anchor- and proposal-free framework.
Source: §1 · p. 2, §3.1 · p. 3
Results
SiamCAR achieves state-of-the-art results across GOT-10K, LaSOT, and other challenging benchmarks; on GOT-10K it ranks first on all indicators and improves over SiamRPN++ by 5.2%, 5.4%, and 9.0% for AO, SR0.5, and SR0.75.
Source: §4.2 · p. 7, §5 · p. 8
Takeaways & Limitations
The framework offers a simple, neat, and generalizable fully convolutional tracker that can be modified with specific modules for further improvement.
Source: §1 · p. 2, §5 · p. 8
Anchor-based trackers remain established high-speed baselines, and their performance is sensitive to relative anchor hyper-parameters and empirical tuning.
Source: §1 · p. 2, §2 · p. 3

Abstract

from arXiv · show

By decomposing the visual tracking task into two subproblems as classification for pixel category and regression for object bounding box at this pixel, we propose a novel fully convolutional Siamese network to solve visual tracking end-to-end in a per-pixel manner. The proposed framework SiamCAR consists of two simple subnetworks: one Siamese subnetwork for feature extraction and one classification-regression subnetwork for bounding box prediction. Our framework takes ResNet-50 as backbone. Different from state-of-the-art trackers like Siamese-RPN, SiamRPN++ and SPM, which are based on region proposal, the proposed framework is both proposal and anchor free. Consequently, we are able to avoid the tricky hyper-parameter tuning of anchors and reduce human intervention. The proposed framework is simple, neat and effective. Extensive experiments and comparisons with state-of-the-art trackers are conducted on many challenging benchmarks like GOT-10K, LaSOT, UAV123 and OTB-50. Without bells and whistles, our SiamCAR achieves the leading performance with a considerable real-time speed.

1. Introduction

Visual tracking remains difficult under unconstrained appearance changes, while existing Siamese approaches rely on matching and often require costly multi-scale processing or careful tuning. SiamCAR instead performs anchor-free, per-pixel classification and regression in an end-to-end Siamese framework, achieving state-of-the-art accuracy and time cost.

Unconstrained tracking is challenged by illumination, scale, clutter, occlusion, and extreme pose variation.Source: §1 · p. 1
Siamese trackers formulate tracking as target matching, but single similarity maps provide limited spatial information and motivate multi-scale search processing.Source: §1 · p. 1
SiamCAR decomposes tracking into per-pixel classification of location labels and regression of a relative bounding box, trained jointly end to end.Source: §1 · p. 2
The framework uses one response map containing category and semantic information to predict object location and bounding box directly.Source: §1 · p. 2
SiamCAR is both anchor- and proposal-free, significantly reducing hyper-parameters and complicated training-time parameter tuning.Source: §1 · p. 2
Without bells and whistles, SiamCAR achieves state-of-the-art tracking performance in both accuracy and time cost.Source: §1 · p. 2

2. Related Works

Prior Siamese trackers improved efficiency through correlation and region proposals, but anchor-based designs remain sensitive to hyper-parameter tuning and fixed shape assumptions. SiamCAR addresses these constraints with a simpler fully convolutional, anchor-free classification-regression framework.

Siamese online-training and offline-tracking methods with deep neural networks have achieved a balance between tracking accuracy and efficiency.Source: §2 · p. 2
SiamFC established a fully convolutional Siamese tracker, followed by models adding correlation filters, dynamic transformations, and region proposal extraction.Source: §2 · p. 2
SiamRPN avoids multi-scale feature extraction by jointly training classification and regression branches for region proposals, improving efficiency.Source: §2 · p. 2
Anchor boxes speed region proposal tracking, but their relative hyper-parameters require careful tuning and empirical tricks for ideal performance.Source: §2 · pp. 2–3
SiamCAR is illustrated as a fully convolutional architecture combining Siamese feature extraction with classification-regression bounding-box prediction.Source: Figure 2 · p. 3
Fixed anchor size and aspect ratio make anchor-based trackers less suitable for objects with large shape deformation and pose variation.Source: §2 · p. 3

3. Proposed Method

SiamCAR extracts a multi-channel Siamese response map and predicts foreground, center-ness, and bounding-box geometry at every location without anchor references. Its tracking phase ranks locations with classification and scale-change scores, then averages the top-k neighboring boxes.

3.1. Feature Extraction with Siamese Subnetwork

The Siamese subnetwork uses shared backbone branches for the template and search region, whose feature maps are combined by channel-by-channel correlation.The resulting response map preserves the channel dimension for downstream classification and regression.Source: §3.1 · pp. 3–4
Multi-layer deep features aggregate low-level visual attributes and high-level semantic information for tracking.The backbone is a modified ResNet-50, and the concatenated feature representation contains 3 × 256 channels.Source: §3.1 · pp. 3–4

3.2. Bounding Box Prediction with Classification and Regression Subnetwork

SiamCAR directly predicts boxes at locations instead of using anchor boxes, enabling fully convolutional end-to-end training without anchor hyper-parameter tuning.A center-ness branch suppresses low-quality predictions from locations far from the object center.Source: §3.2 · p. 4
Each response-map location is classified as foreground or background and regressed to four distances from that location to the bounding-box sides.The regression vector is t(i, j) = (l, t, r, b), while classification outputs a two-dimensional foreground/background score.Source: §3.2 · p. 4
The training objective combines classification, center-ness, and regression losses, with empirically selected weights λ1 = 1 and λ2 = 3.Classification uses cross-entropy, while bounding-box regression uses IOU loss.Source: §3.2 · p. 4

3.3. The Tracking Phase

During tracking, SiamCAR selects the highest-scoring location, then averages the top 3 boxes among 8 neighboring points for stable predictions.Neighboring points are ranked using clsij × pij, where pij is a scale-change penalty; k = 3 was found empirically to provide stable results.Source: §3.3 · p. 5

4. Experiments

SiamCAR is evaluated on GOT-10K, LaSOT, OTB-50, and UAV123 using the specified training and testing protocols. It achieves leading accuracy across these benchmarks while maintaining real-time speed.

Results on GOT-10K

5.2%, 5.4%, and 9.0% relative gains over SiamRPN++ are reported for GOT-10K AO, SR0.5, and SR0.75, respectively.SiamCAR ranks first on all reported GOT-10K indicators, under a protocol using the same training data and unseen test classes.Source: §4.2 · p. 7

Results on LaSOT

3.1%, 1.9%, and 1.1% relative gains over SiamRPN++ are reported for LaSOT’s three indicators.Against the provided baselines, the reported relative improvements exceed 14%, 13.7%, and 11%.Source: §4.3 · p. 7

Results on OTB-50

SiamCAR achieves the best reported results on OTB-50’s listed challenging aspects and improves accuracy especially for low resolution, out-of-plane rotation, and background clutter.The authors connect these results to semantic information implicitly decoded by the classification-regression subnetwork.Source: §4.4 · pp. 7–8

Results on UAV123

SiamCAR outperforms the other trackers on both UAV123 indicators while using a simpler network and no heuristic tuning parameters.The comparison includes nine state-of-the-art approaches, including SiamRPN++, SiamRPN, SiamFC, and ECO.Source: §4.5 · p. 8

Run-time evaluation

52.27 FPS is reported for SiamCAR on GOT-10K, indicating real-time evaluation speed.The speed is measured on a machine with one RTX 2080ti.Source: §4.6 · p. 8

5. Conclusions

SiamCAR presents a simple Siamese classification-regression framework that solves visual tracking per pixel and achieves state-of-the-art results across challenging benchmarks.

SiamCAR end-to-end trains a deep Siamese network for visual tracking using a fully convolutional framework.Source: §5 · p. 8
The framework decomposes tracking into per-pixel classification and bounding-box regression.Source: §5 · p. 8
SiamCAR achieves state-of-the-art results on GOT-10K and other challenging benchmarks without bells and whistles.Source: §5 · p. 8
Its state-of-the-art results on LaSOT demonstrate generalizability across a large dataset.Source: §5 · p. 8
The simple and neat framework can be modified with specific modules for further improvement.Source: §5 · p. 8