Source-linked AI summary
SiamMask: A Framework for Fast Online Object Tracking and Segmentation
Weiming Hu, Qiang Wang, Li Zhang, Luca Bertinetto, Philip H. S. Torr
TL;DR
Tracking boxes are quick to initialize but do not describe object contours, while video object segmentation has often been too slow for real-time use. SiamMask adds a binary-segmentation task to offline-trained Siamese tracking, producing masks online at 55 frames per second and performing competitively across tracking and VOS benchmarks.
Problem
Bounding boxes poorly represent object contours, while traditional video object segmentation methods have often been slow and even the fastest had not operated in real time.
Source: §1 · p. 1Method
SiamMask augments fully-convolutional Siamese networks with a class-agnostic binary-segmentation task alongside target similarity and bounding-box regression.
Source: §1 · pp. 1–2Results
SiamMask runs at 55 frames per second, establishes state-of-the-art real-time tracking, and is competitive across VOS benchmarks while being the fastest by a large margin.
Source: §1 · p. 2Takeaways & Limitations
With one bounding-box initialization and no test-sequence updates, SiamMask produces masks and rotated boxes online and extends to multiple-object tracking and segmentation through a cascade.
Source: §1 · p. 2Extreme motion blur can make SiamMask particularly susceptible because it is supervised and trained offline only, while labels are expected to contain noise.
Source: §6.6 · p. 12
Abstract
from arXiv · showhide
In this paper we introduce SiamMask, a framework to perform both visual object tracking and video object segmentation, in real-time, with the same simple method. We improve the offline training procedure of popular fully-convolutional Siamese approaches by augmenting their losses with a binary segmentation task. Once the offline training is completed, SiamMask only requires a single bounding box for initialization and can simultaneously carry out visual object tracking and segmentation at high frame-rates. Moreover, we show that it is possible to extend the framework to handle multiple object tracking and segmentation by simply re-using the multi-task model in a cascaded fashion. Experimental results show that our approach has high processing efficiency, at around 55 frames per second. It yields real-time state-of-the-art results on visual-object tracking benchmarks, while at the same time demonstrating competitive performance at a high speed for video object segmentation benchmarks.
1. Introduction
SiamMask combines tracking and segmentation to retain online Siamese tracking speed while providing a more detailed target representation. It extends the same multi-task framework to multiple-object tracking and segmentation.
1. Introduction
- Bounding boxes offer quick initialization but poorly approximate contours under complex motion and non-rigid deformation, while pixel masks provide detailed representations at greater computational cost.Traditional VOS methods were often slow, and even the fastest approaches had not operated in real time.Source: §1 · p. 1
- SiamMask jointly trains a fully-convolutional Siamese network for target similarity, bounding-box regression, and class-agnostic binary segmentation.Segmentation labels are needed offline to compute the segmentation loss, not during online tracking or segmentation.Source: §1 · pp. 1–2
- At 55 frames per second on a consumer-grade GPU, SiamMask achieves state-of-the-art real-time tracking and competitive, fastest-by-a-large-margin performance among VOS approaches.It uses a single bounding-box initialization, operates online without updates, and avoids costly VOS techniques such as fine-tuning, data augmentation, and optical flow.Source: §1 · p. 2
- A two-stage cascade extends the framework to multiple-object tracking and segmentation, placing second at the YouTubeVIS challenge.The first stage locates each target and the second extracts a pixel-wise mask; the Hungarian algorithm handles data association.Source: §1 · p. 2
2. Related Work
The related work contrasts fast online tracking with pixel-accurate but computationally demanding video object segmentation. SiamMask is positioned as combining online tracking and detailed masks without giving up bounding-box initialization.
2.1. Visual object tracking
- Fully-convolutional Siamese trackers train similarity functions offline on frame pairs and evaluate them once per frame, with later work adding proposals, hard-negative mining, ensembles, memory, and cascaded regression.Other cited directions include meta-learning, distractor handling, graph attention, salient-region mining, architecture search, adversarial robustness, and self-supervised learning.Source: §2.1 · p. 2
2. Related Work
- Tracking benchmarks favor sequential, online processing and speed, whereas VOS has prioritized accurate object representations.The discussion notes that fast online trackers typically use boxes while accurate-mask methods often sacrifice speed and online operation.Source: §2.2 · p. 3, §2.3 · p. 3
2.2. Video object segmentation (VOS)
- VOS methods use mask propagation or frame-wise processing, often adding fine-tuning, data augmentation, or optical flow at the cost of speed and online operation.OSMN and RGMP, two fast approaches, still run below 10 frames per second.Source: §2.2 · p. 3
2.3. Tracking and segmentation
- Prior joint tracking-and-segmentation methods include online mask trackers and box-initialized VOS, but some are slow or require online learning.The fastest variant of Yeo et al.'s superpixel tracker runs at 4 frames per second; other cited methods use online learning that can impact speed.Source: §2.3 · p. 3
3. Fully-convolutional Siamese Networks
Fully-convolutional Siamese networks compare an exemplar with a search region to localize a target using a dense response map. SiamFC classifies candidate locations, while SiamRPN adds anchor-based classification and box regression.
3. Fully-convolutional Siamese Networks
- The fully-convolutional Siamese framework is adopted to support online operation and fast speed, starting from SiamFC and SiamRPN.The paper introduces these networks before describing its approach.Source: §3 · p. 3
3.1. SiamFC
- SiamFC processes exemplar and search crops with the same CNN and cross-correlates their features into a dense response map whose maximum indicates the target location.The paper replaces simple cross-correlation with depth-wise cross-correlation to let each response encode richer target information.Source: §3.1 · p. 3
3.2. SiamRPN
- SiamRPN gives each candidate window anchor proposals with object/background scores and corresponding bounding-box predictions.The classification branch uses cross-entropy; the regression branch uses smooth L1 loss with normalized coordinates.Source: §3.2 · p. 4
4. SiamMask
SiamMask extends fully-convolutional Siamese trackers with a segmentation task, so the network can produce pixel masks alongside tracking outputs. Its sections describe the branches, mask refinement and loss, box generation, and training and testing procedures.
4.1. Multi-branch network architecture
- SiamMask adds a segmentation branch to SiamFC and SiamRPN; its two-branch variant predicts scores and masks, while its three-branch variant also regresses boxes.Depth-wise cross-correlation produces multi-channel response maps for the candidate regions.Source: §4.1 · p. 4
4.2. Mask representation and refinement
- Masks begin from a flattened candidate-region representation, then refinement modules combine it with features from other network depths and progressively upsample it.The U3 example combines representations from two convolutional paths before producing a 2×-upscaled mask representation.Source: §4.2 · p. 5
4.3. Loss function
- A pixel-wise binary logistic loss trains the mask branch on positive candidate regions and is combined with the score loss, plus box-regression loss in the three-branch variant.The authors report that including negative regions in a weighted mask loss performed worse than considering positive regions only.Source: §4.3 · p. 5
4.4. Box generation
- Three strategies convert predicted masks to tracking boxes: an axis-aligned enclosing rectangle, a rotated minimum bounding rectangle, and the VOT optimization strategy.The paper empirically compares these alternatives in Table 4.Source: §4.4 · p. 5
4.5. Training and testing
- At test time, offline-trained SiamMask starts from an axis-aligned box, selects the mask at the top-scoring classification location, and updates no network parameters.Training labels a three-branch candidate positive at anchor-box IoU ≥ 0.6; the two-branch variant uses a center-distance threshold of 16 pixels in feature space.Source: §4.5 · pp. 5–6
5. Multiple Object Tracking and Segmentation
For multiple-object tracking and segmentation, SiamMask is used in a cascade for each object, with mask-based assignment associating detections to existing trajectories. The assignment also supports tracking objects that appear or disappear.
5. Multiple Object Tracking and Segmentation
- Unlike single-object tracking, the multi-object task must disambiguate instance identities across frames, a particular challenge in crowded scenes.Source: §5 · p. 6
- Detections are assigned to predicted trajectory masks using pairwise mask IoU as affinity, with at most one-to-one matches solved by the Hungarian algorithm.The paper formulates identity association as an optimal transport problem.Source: §5 · p. 6
- A detector supplies candidate masks for initialization; per object, SiamMask first predicts a coarse location, then refines a mask from a crop around the highest-scoring box.The refined masks are associated with current trajectories, and the assignment accounts for newly appeared or disappeared objects.Source: §5 · pp. 6–7
6. Experiments
The experiments evaluate SiamMask on tracking and video-segmentation benchmarks, examining its speed and accuracy, mask-derived tracking boxes, and bounding-box-only initialization. They also assess challenge performance, two-stage strategy and ablations, and qualitative examples of strengths and limitations.
6.2. Evaluation for tracking
- On VOT-2016, SiamMask-MBR reaches mAP@0.5 IOU 85.4, improving by 29 and 9.2 points over SiamFC and SiamRPN; its mAP@0.7 gains are 41.6 and 18.4 points.SiamMask-Opt has the highest IOU and mAP but requires significant computation; MBR offers an advantage over axis-aligned boxes while retaining real-time speed.Source: Table 4 · p. 8, §6.2 · pp. 8–9
6.1. Implementation
- On one NVIDIA RTX 2080, SiamMask runs at 55 frames per second (two-branch) and 60 (three-branch) without test-sequence adaptation; inference selects the highest-scoring mask and binarizes it at 0.5.The two-branch variant uses the mask-derived Min-max box to crop the next search region; the three-branch variant uses its highest-scoring box-branch output.Source: §6.1 · p. 8
6.2. Evaluation for tracking
- EAO 0.380 at 55 frames per second: SiamMask’s three-branch MBR variant significantly outperforms DaSiamRPN on VOT-2018; the two-branch variant reaches EAO 0.334.SiamMask-Opt achieves the best overall performance but incurs significant computational cost.Source: Table 6 · p. 9, §6.2 · p. 9
- SiamMask obtains the best results for most VOT-2016 and VOT-2018 scene attributes, with pixel-wise masks supporting accurate handling of rapid non-rigid deformations.The evaluated attributes include occlusion, illumination and motion changes, scale changes, and camera motion.Source: §6.2 · p. 9, Figure 9 · p. 10
- Average overlap rises by 37% and success rate by up to 150% over CFNet on GOT-10k; on TrackingNet, SiamMask outperforms competitors across metrics and exceeds ATOM by 2.1%.The GOT-10k comparison is not fully like-for-like because training-set class overlap was not enforced for SiamMask.Source: Table 8 · p. 10, §6.2 · p. 10, Table 9 · p. 10
6.3. Evaluation for video object segmentation (VOS)
- On DAVIS-2016, SiamMask has accuracy comparable to fast VOS methods at significantly higher speed; OnAVOS [32] and MSK achieve the best quality but are hundreds of times slower.SiamMask uses a bounding box rather than a pixel-wise mask for initialization and does not update its backbone online.Source: Figure 10 · p. 11, §6.3 · p. 11, Table 11 · p. 11
- Across DAVIS-2016, DAVIS-2017, and YouTube-VOS, SiamMask achieves competitive online segmentation at fast speed, with remarkably low temporal decay, best seen-class accuracy on YouTube-VOS, and nearly two orders faster than OnAVOS [32] and SFL.On DAVIS-2017, its temporal robustness is surpassed only by the slower FAVOS.Source: §6.3 · p. 11
- The experiments present SiamMask as a strong baseline for online video object segmentation as well as tracking.Source: §6.3 · p. 12
6.4. Evaluation for multiple object tracking and segmentation
- The two-stage variant improves YouTube-VIS validation results by 1.5% in mAP and 1.6% in average recall over the single-stage variant.The authors attribute the gains to the regression branch limiting the area segmented, reducing difficulty for the mask branch.Source: §6.4 · p. 12, Table 14 · p. 12
- SiamMask ranks second on the 2019 YouTube-VIS leaderboard and improves mAP by a relative 46% over the organizers’ official baseline.The second-place approach treated detection, classification, segmentation, and tracking as separate problems.Source: §6.4 · p. 12, Table 15 · p. 12
6.5. Ablation studies
- Ablations show that ResNet-50 improves performance at a reasonable speed cost, two- and three-branch variants beat their respective baselines, and mask refinement improves contour accuracy but not EAO significantly.With the mask branch unused at inference, two-branch EAO rises from 0.251 to 0.265 and three-branch EAO from 0.359 to 0.363 versus their counterparts; the authors caution that this setup is too limited to establish a regularization effect confidently.Source: §6.5 · p. 12
6.6. Qualitative examples
- Failure cases include extreme motion blur, overlapping object trajectories that cause identity confusion, and initialization on a texture or object part rather than an object.The authors suggest modeling relationships between pixels within a mask to address confusion, but note that this would reduce tracking speed.Source: §6.6 · pp. 12–13, §6.4 · p. 12
- Qualitative results show SiamMask produces accurate masks with fast-moving objects despite distractors, and on sequences with large illumination variations and severe motion blur; it handles changes in scale, view angle, shape, and minor occlusion.For YouTube-VIS, positive results are shown on challenging videos involving deformation, occlusion, or rapid motion.Source: §6.6 · p. 13
7. Conclusion
SiamMask enables fully-convolutional Siamese trackers to produce class-agnostic binary masks, performing visual object tracking and semi-supervised video object segmentation. It also extends to multiple-object tracking and segmentation by cascading two models.
7. Conclusion
- SiamMask produces class-agnostic binary segmentation masks and performs visual object tracking and semi-supervised video object segmentation, with better accuracy than most real-time trackers and the fastest speed among VOS methods.Source: §7 · p. 14
- The framework extends to multiple-object tracking and segmentation by cascading two models.Source: §7 · p. 14