Source-linked AI summary
Siam R-CNN: Visual Tracking by Re-Detection
Paul Voigtlaender, Jonathon Luiten, Philip H. S. Torr, Bastian Leibe
TL;DR
Visual tracking must handle distractors, drift, and objects that disappear and reappear. Siam R-CNN uses Siamese full-image re-detection, hard-example mining, and tracklet dynamic programming to model target and distractor histories. It outperforms previous methods on ten tracking benchmarks, especially in long-term tracking, while also supporting box-only video object segmentation.
Problem
Visual tracking is challenged by similar-looking distractors, tracker drift, and long-term disappearance and reappearance.
Method
Siam R-CNN combines a Siamese two-stage full-image re-detector with hard-example mining and Tracklet Dynamic Programming over target and distractor tracklets.
Results
Siam R-CNN outperforms previous methods on ten tracking benchmarks, with especially strong results for long-term tracking.
Takeaways & Limitations
The approach supports long-term re-detection after disappearance and extends box tracking to video object segmentation using a second mask-producing network.
Takeaways & Limitations
Even the fastest variant is not real-time, and TDPA limits temporal gaps between tracklets to 1500 frames for efficiency.
Abstract
from arXiv · showhide
We present Siam R-CNN, a Siamese re-detection architecture which unleashes the full power of two-stage object detection approaches for visual object tracking. We combine this with a novel tracklet-based dynamic programming algorithm, which takes advantage of re-detections of both the first-frame template and previous-frame predictions, to model the full history of both the object to be tracked and potential distractor objects. This enables our approach to make better tracking decisions, as well as to re-detect tracked objects after long occlusion. Finally, we propose a novel hard example mining strategy to improve Siam R-CNN's robustness to similar looking objects. Siam R-CNN achieves the current best performance on ten tracking benchmarks, with especially strong results for long-term tracking. We make our code and models available at www.vision.rwth-aachen.de/page/siamrcnn.
1. Introduction
Siam R-CNN applies full-image Siamese re-detection to visual tracking and combines it with hard-example mining and tracklet dynamic programming. The approach targets distractors, drift, long occlusions, and strong performance across tracking tasks.
- Siam R-CNN: Siam R-CNN adapts Faster R-CNN into a Siamese re-detector that locates a template object anywhere in an image and regresses its bounding box.Region proposals are aligned to a common size, improving robustness to object-scale and aspect-ratio changes.
- Motivation: Distractor objects challenge re-detection, while spatial priors and online adaptation can cause tracker drift.The paper addresses this problem with hard-example mining and tracklet-based inference.
- Tracklet Dynamic Programming: Tracklet Dynamic Programming models target and distractor histories by grouping detections over time and selecting the best object using dynamic programming.It iteratively updates tracking history with a small number of new re-detections per timestep.
- Efficiency: 4.7 FPS is achieved by the full system, while a speed-optimized variant exceeds 15 FPS.The full system prioritizes accuracy, especially for long-term tracking, whereas the fastest variant remains non-real-time.
- Results: Siam R-CNN outperforms previous methods on six short-term and four long-term tracking benchmarks, with gains up to 10 percentage points in long-term tracking.Using an off-the-shelf box-to-segmentation network also yields stronger results on four recent VOS benchmarks using only first-frame bounding boxes.
2. Related Work
Related tracking methods use online adaptation, local search, or specialized Siamese designs, while Siam R-CNN searches the whole image with a two-stage re-detector. The method is also adapted to video object segmentation when only a template bounding box is available.
- Visual Object Tracking: Visual object tracking follows an object through a video from its first-frame bounding box, with long-term benchmarks adding disappearance and reappearance.This defines the standard and more difficult long-term tracking settings.
- Tracking by Re-Detection: Classical trackers commonly learn classifiers online, whereas Siam R-CNN learns expected appearance variations through offline training.The contrast concerns how appearance models are obtained.
- Siamese Tracking: Siamese region proposal trackers use single-stage anchor classification or local-window search, while Siam R-CNN uses two-stage proposals and whole-image re-detection.Two-stage detection first generates proposals, then aligns and classifies their features.
- Long-Term Tracking: Long-term tracking methods often enlarge the search window when confidence is low, whereas Siam R-CNN searches over the whole image.The paper reports stronger results across many benchmarks for this two-stage Siamese re-detector.
- Video Object Segmentation: Siam R-CNN is adapted to video object segmentation by using a second network to produce masks from box-tracking results.This addresses the setting where only the template bounding box, not the first-frame mask, is provided.
3. Method
Siam R-CNN combines a Siamese two-stage re-detector with hard negative mining and tracklet-based dynamic programming. The method re-detects candidate objects, builds temporally consistent tracklets, and selects among targets and distractors using spatial and temporal evidence.
- Siam R-CNN: Siam R-CNN adapts Faster R-CNN into a Siamese two-stage re-detector that compares proposed regions with a first-frame template.Region proposals are aligned before comparison, supporting changes in object size and aspect ratio.
- Siam R-CNN: The re-detection head concatenates RoI-aligned features from each proposal and the initialization box, then classifies the proposal as the reference object or not.The concatenated features pass through a 1 × 1 convolution before a three-stage cascade re-detection head.
- Video Hard Example Mining: Hard example mining retrieves visually similar objects from other videos and adds sampled nearest neighbors as negative training examples.An embedding index retrieves 10,000 nearest-neighbor boxes, from which 100 are sampled for each training step.
- Tracklet Dynamic Programming Algorithm: TDPA maintains short tracklets for the target and similar-looking distractors, using their complete histories to select the most likely object over time.Dynamic programming scores sequences of tracklets, while re-detections from the first-frame template and previous-frame detections provide the tracking evidence.
- Tracklet Dynamic Programming Algorithm: Tracklets extend only when similarity exceeds α and neither competing detection nor tracklet is within β of the best match; ambiguous detections start new tracklets.Bounding-box spatial distance is computed from normalized center, width, and height coordinates using the L∞ norm.
4. Experiments
Siam R-CNN is evaluated across short-term, long-term, and video object segmentation benchmarks using one shared TDPA hyper-parameter setting. It achieves strong benchmark performance, while ablations and generic-object analysis examine accuracy, speed, and proposal recall.
- Evaluation protocol: A single TDPA hyper-parameter set, tuned on DAVIS 2017 training data, is used across all reported benchmarks.The re-detector was not trained on that set, and parameters were not separately tuned per benchmark.
- Short-term tracking: 70.1% AUC on OTB2015 equals the previous best, while TrackingNet reaches 81.2% success, 7.2 percentage points above DiMP-50.On GOT-10k, Siam R-CNN reaches 64.9% success, 3.8 percentage points above the previous best under restricted training data.
- Long-term tracking: Siam R-CNN significantly outperforms previous methods on LTB35, LaSOT, and OxUvA long-term tracking benchmarks.On OxUvA, its MaxGM is more than 10 percentage points higher than all previous methods; long-term sequences include repeated disappearances and reappearances.
- Ablation and timing analysis: Hard example mining improves results by up to 1.7 percentage points, and TDPA outperforms both Argmax and the short-term variant on all three ablation datasets.The full configuration runs at 4.7 FPS with ResNet-101, 1000 RPN proposals, and TDPA.
- Generic object tracking analysis: Adding up to 100 previous-frame re-detections raises unknown-object RPN recall from 69.1% to 95.5% with 1000 proposals, avoiding the approximately 1 FPS speed of 10,000 proposals.Known-object recall with 1000 proposals is 98.2%.
5. Conclusion
Siam R-CNN combines Siamese full-image re-detection with tracklet dynamic programming and achieves strong results across tracking benchmarks, especially for long-term tracking.
- Siam R-CNN combines a Siamese two-stage full-image re-detection architecture with a Tracklet Dynamic Programming Algorithm.
- The method outperforms previous methods on ten tracking benchmarks, with especially strong results for long-term tracking.
Abstract
The supplementary material provides additional methodological details and analyses covering training, hard example mining, short-term tracking, rotated boxes, speed, data, and attribute-specific performance.
- The supplementary material details Siam R-CNN training, video hard example mining, short-term tracking, and rotated bounding box estimation.
- It also analyzes speed improvements, training-data usage, and per-attribute variants on OTB2015.
A.1. Training
The supplementary experiments describe Siam R-CNN’s training and short-term tracking procedures, then evaluate rotated boxes, speed, backbone effects, and data usage.
- Training: Training resizes images while preserving aspect ratio, uses multi-stage learning-rate schedules, and applies hard-example training after the main training phase.Images are scaled between 640 and 800 pixels on the small edge, capped at 1333 pixels on the long edge; training uses two GPUs for 1 million steps followed by additional schedules.
- Training: Jittered ground-truth boxes and retrieved nearest-neighbor boxes expose the network to imperfect regions and difficult examples during training.Gaussian noise is added before cascade feature pre-computation, while nearest-neighbor boxes provide additional examples and positive boxes from randomly selected reference-video frames balance negatives.
- Short-term Tracking: The short-term tracker adds shifted versions of the previous prediction as RoIs before Siamese re-detection.This exploits the known proximity of the target to the previous prediction and compensates for possible RPN false negatives.
- Efficiency and Data: 62.3 on LaSOT versus 56.8 for DiMP-50 is obtained with a ResNet-50 backbone, while reduced configurations reach 15.2 frames per second.Using 100 RoIs nearly doubles speed with small score changes, and the fastest setup retains strong long-term tracking results.
- Efficiency and Data: Siam R-CNN uses less training data than several recent methods despite using more video datasets.DiMP-50 and ATOM use 2.28 times more videos, while SiamRPN++ uses 21.3 times more.
B.3. Per-Attribute Analysis
The supplementary analysis examines attribute-specific performance, disappearance handling, segmentation fine-tuning, and generalization across additional benchmarks.
- Per-Attribute Ablation: Hard example mining improves OTB2015 success across all attributes, especially low resolution and background clutter.
- Per-Attribute Ablation: TDPA significantly outperforms the short-term variant on out-of-view sequences by maintaining robust spatio-temporal consistency cues during disappearance.TDPA tracks potential objects simultaneously and selects likely tracklets using dynamic programming.
- Segmentation Fine-Tuning: 300 Box2Seg fine-tuning steps produce 74.8 J &F at 1 FPS, compared with 70.6 J &F at 3.1 FPS without fine-tuning.
- Further Benchmarks: The tracker is evaluated on eight additional benchmarks using the same parameters except for the three VOT benchmarks.These include five short-term tracking, one long-term tracking, and two video object segmentation benchmarks.
C.1. Further Short-Term Tracking Evaluation
Additional short-term evaluations show that Siam R-CNN remains competitive across several established benchmarks, with particularly strong results on OTB-50 and VOT2015/2016.
- OTB-50: 66.3 AUC on OTB-50 exceeds ACT’s previous best published result by 0.6 percentage points.
- OTB2013: 70.4 AUC on OTB2013 is comparable to state-of-the-art trackers but 1.8 percentage points below GFS.
- Evaluation scope: The supplemental evaluation covers OTB-50, OTB2013, VOT2015, VOT2016, and TempleColor128 in addition to the main-paper benchmarks.
- VOT2015: The short-term version achieves 45.4 EAO on VOT2015, surpassing DaSiamRPN by 0.8 percentage points.
- VOT2016: The short-term version achieves 46.5 EAO on VOT2016 and trails only the results of Upda
C.3. Further Video Object Segmentation Evaluation
Further segmentation experiments evaluate Siam R-CNN on DAVIS and YouTube-VOS, emphasizing performance when only first-frame bounding boxes are available and the speed–accuracy trade-off.
- Evaluation details: The DAVIS evaluations report timing conventions and identify a faster Siam R-CNN configuration using ResNet-50, half resolution, and 100 RPN RoIs.
- DAVIS 2016: 78.6% J &F on DAVIS 2016 is the strongest result among methods using only the first-frame bounding box, 8.8 points above SiamMask.
- DAVIS 2016: 87.1% J &F on DAVIS 2016 with Box2Seg is close to STM-VOS’s best result of 89.3%.
- DAVIS 2017: 53.3% J &F on the DAVIS 2017 test-dev set exceeds SiamMask by more than 10 percentage points, while STM-VOS reaches 72.3% using first-frame masks.
- YouTube-VOS 2018: On YouTube-VOS 2018, Siam R-CNN offers a good speed–accuracy trade-off and is beaten only by STM-VOS, which relies on the first-frame mask.
C.5. Thorough Comparison to Previous Methods
A unified comparison across 11 short-term and four long-term benchmarks finds Siam R-CNN broadly competitive, with especially consistent gains in long-term tracking.
- Comparison protocol: Table 21 compares trackers from major vision conferences in 2018 and 2019 using results from original papers, later papers, or open-source implementations.
- Evaluation scope: The comparison includes 11 short-term and four long-term tracking benchmarks presented across the main paper and supplemental material.
- Short-term tracking: Siam R-CNN outperforms all previous methods on six of 11 short-term benchmarks, sometimes by up to 7.2 percentage points.
- Short-term tracking: On the remaining five short-term benchmarks, Siam R-CNN is close to the best results, with only a few methods ahead by modest margins.
- Long-term tracking: Across all four long-term benchmarks, Siam R-CNN outperforms previous methods by between 3.9 and 10.1 percentage points.