Source-linked AI summary
SPM-Tracker: Series-Parallel Matching for Real-Time Visual Object Tracking
Guangting Wang, Chong Luo, Zhiwei Xiong, Wenjun Zeng
TL;DR
Visual tracking must jointly handle robustness to appearance changes and discrimination against clutter or similar objects, a difficult balance for existing approaches. SPM-Tracker separates these goals into coarse and fine matching within a series-parallel SiamFC-based design, achieving strong benchmark performance at 120fps.
Problem
Visual tracking must balance robustness to target appearance changes with discrimination from clutter and similar objects.
Method
SPM-Tracker uses coarse matching for robustness and fine matching for discrimination, with coarse proposals feeding fine matching and both stages’ scores and box refinements fused.
Results
SPM-Tracker achieves an AUC of 0.687 on OTB-100, EAOs of 0.434 and 0.338 on VOT-16 and VOT-17, respectively, and runs at 120fps on a NVIDIA P100 GPU.
Takeaways & Limitations
The series-parallel framework combines generalized training in CM with a distance-learning relation network in FM.
Takeaways & Limitations
The tracker can drift when a target is occluded by a similar object and may track an entire object when the template contains only part of it.
Abstract
from arXiv · showhide
The greatest challenge facing visual object tracking is the simultaneous requirements on robustness and discrimination power. In this paper, we propose a SiamFC-based tracker, named SPM-Tracker, to tackle this challenge. The basic idea is to address the two requirements in two separate matching stages. Robustness is strengthened in the coarse matching (CM) stage through generalized training while discrimination power is enhanced in the fine matching (FM) stage through a distance learning network. The two stages are connected in series as the input proposals of the FM stage are generated by the CM stage. They are also connected in parallel as the matching scores and box location refinements are fused to generate the final results. This innovative series-parallel structure takes advantage of both stages and results in superior performance. The proposed SPM-Tracker, running at 120fps on GPU, achieves an AUC of 0.687 on OTB-100 and an EAO of 0.434 on VOT-16, exceeding other real-time trackers by a notable margin.
1. Introduction
Visual tracking must balance robustness to appearance changes with discrimination against clutter and similar objects, while deep-feature online training creates speed challenges. SPM-Tracker separates these goals into coarse and fine matching stages connected in a series-parallel structure.
- 1. Introduction: Visual tracking requires robustness to illumination, motion, viewpoint, and deformation, while also discriminating targets from clutter and similar objects.
- 1. Introduction: Deep-feature online training is extremely time consuming, limiting the speed of high-performance trackers.
- 1. Introduction: SiamFC-based tracking seeks an offline-trained embedding space that is simultaneously robust and discriminative, but balancing both requirements in one network is difficult.
- 1. Introduction: SPM-Tracker uses coarse matching to enhance robustness and fine matching to improve discrimination within a two-stage SiamFC-based network.
- 1. Introduction: Generalized training strengthens coarse-stage robustness, while distance learning promotes fine-stage discrimination; matching scores and box refinements are fused in the series-parallel design.
2. Related Work
Related work progresses from adaptive discriminative and correlation-filter trackers toward static SiamFC-based methods that avoid slow deep-feature online training. Subsequent SiamRPN-style and two-stage designs address scale, shape, proposal generation, and stronger matching.
- 2. Related Work: Discriminative trackers commonly learn a decision boundary between the object and background, with adaptive methods updating the classifier during tracking.
- 2. Related Work: Correlation-filter trackers support adaptation and scale estimation, but deep features make their indispensable online training extremely slow.
- 2. Related Work: SiamFC uses Siamese CNN features and cross-correlation for dense, efficient sliding-window evaluation, enabling static high-performance tracking with expressive deep features.
- 2. Related Work: SiamFC-based methods struggle to learn embeddings that are both robust and discriminative, motivating distractor-aware training and two-branch approaches.
- 2. Related Work: SiamRPN introduces region proposals for bounding-box size adjustment and removes the need for multi-scale testing, while two-stage designs can provide stronger matching.
3. Our Approach
SPM-Tracker uses separate coarse and fine matching stages to emphasize robustness and discrimination, connecting them in series for proposal generation and in parallel through score and box fusion.
- Series-Parallel Matching Framework: The SPM framework uses Siamese features followed by coarse and fine matching stages organized in a series-parallel structure.The framework is designed for robust and discriminative visual object tracking.
- Series-Parallel Matching Framework: The CM stage emphasizes robustness by proposing target candidates despite drastic appearance changes, while the FM stage refines discrimination using top-scoring proposals.FM evaluates regional features from a limited set of CM proposals to distinguish the target from similar objects.
- Coarse Matching Stage: Generalized training treats image pairs containing different objects from the same category as positive samples for the CM stage.This extends conventional same-instance positive sampling to improve robustness to appearance variation.
- Coarse Matching Stage: The CM stage generates strong responses under severe target deformation, whereas SiamRPN barely responds and SiamFC localizes less precisely.The comparison is reported for response maps in Figure 4.
- Fine Matching Stage: The FM stage uses a lightweight relation network instead of cross-correlation to learn a more powerful, complementary similarity measure from concatenated proposal features.The network produces feature embeddings for classification and box regression.
- Series-Parallel Matching Framework: The two stages fuse similarity scores and box deltas by weighted sum, combining high CM scores for same-category objects with high FM scores for the true target.This fusion is intended to combine robustness with discrimination power.
4. Implementation
The implementation uses an end-to-end AlexNet-based network with joint classification and box-regression losses in both stages, followed by proposal processing and weighted inference fusion.
- Network Structure and Parameters: The tracker uses a standard AlexNet backbone with padded feature maps for RoI Align and extracts fused shallow and deep regional features for FM proposals.The target patch is 127 × 127 × 3, and FM regional features are pooled to 6 × 6 × 640.
- Training: The network is trained end-to-end using classification and box-regression losses for both CM and FM stages.The overall loss combines four terms, with λ2 = 2 and the other listed weights equal to 1.
- Training: CM labels use IoU thresholds above 0.6 for positives and below 0.3 for negatives, while FM uses thresholds above or below 0.5.Intermediate-overlap patches are ignored for CM, and box regression loss applies to positive samples in both stages.
- Training: Training pairs combine video and still-image datasets, sampled as videos to still images at 4 : 1 and same-instance, same-category, and different-category pairs at 2 : 1 : 1.The listed datasets include VID, Youtube-BB, COCO, ImageNet DET, Cityperson, and WiderFace.
- Inference: During inference, the CM stage produces boxes, reserves the candidate with greatest overlap to the previous tracking box, and processes other candidates with score filtering and NMS.The search image is cropped around the previous box and resized to 271 × 271.
- Inference: The final score and box coordinates are weighted sums of CM and FM outputs, and the tracker runs at 120fps on a single NVIDIA P100 GPU.The final highest-scoring candidate is selected after cosine-window processing.
5. Experiments
Experiments analyze the series-parallel design, generalized training, candidate selection, feature fusion, and relation-network fine matching. Across OTB, VOT, and large-scale benchmarks, SPM-Tracker delivers strong real-time tracking performance, while qualitative cases reveal specific failure modes.
- 5.1. Analysis of the Series-Parallel Structure: The series-parallel fusion scheme significantly outperforms CM-only and CM+FM Series alternatives.Fusion combines stages focused on robustness and discrimination power.
- 5.2. Analysis of the CM Stage: Generalized training improves CM performance across all three benchmark datasets.It uses same-category image pairs containing different objects as positive training samples.
- 5.2. Analysis of the CM Stage: Recall increases with candidate count before saturating at approximately K = 9, while generalized training further boosts recall at saturation.Using multiple candidates is substantially better than using a single candidate box.
- 5.3. Analysis of the FM Stage: 0.687 AUC is achieved on OTB-100 with conv2 + conv4 fusion, exceeding single-layer alternatives.Single-layer conv2, conv3, and conv4 models achieve AUC scores of 0.666, 0.675, and 0.676, respectively.
- 5.4. Comparison with State-of-the-Arts: SPM-Tracker outperforms other real-time trackers across OTB benchmarks and balances tracking performance with inference speed.The OTB comparison covers OTB 2013, OTB-50, and OTB-100.
- 5.4. Comparison with State-of-the-Arts: On VOT-16 and VOT-17, SPM-Tracker is the best-performing real-time tracker by accuracy and EAO, while also performing favorably on TrackingNet and LaSOT.Removing two extra training datasets lowers OTB-100 AUC to 0.671 and VOT-16 EAO to 0.432.
- 5.5. Qualitative Results: Successful sequences show robustness to large deformation and more precise box locations through two-stage refinement.The tracker handles challenging jump and diving sequences where some competing trackers fail.
- 5.5. Qualitative Results: Failures occur when targets are occluded by similar objects or when the ground-truth target is only part of a larger object.The tracker may drift under similar-object occlusion and may follow the entire object because of its objectness bias.
6. Conclusion
The paper presents SPM-Tracker, a static discriminative tracker using a novel series-parallel structure for two-stage matching. OTB and VOT evaluations show superior tracking performance, while future work targets occlusion-induced drift.
- 6. Conclusion: SPM-Tracker uses a novel series-parallel structure for two-stage matching.The tracker is presented as a static discriminative tracker.
- 6. Conclusion: Evaluations on OTB and VOT benchmarks show superior tracking performance.The conclusion identifies these benchmarks as evidence for the tracker’s performance.
- 6. Conclusion: Future work will explore template update and forward-backward verification to address drifting when targets are occluded by similar objects.The authors describe these as possible choices rather than established solutions.