PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Online Object Tracking and Segmentation: A Unifying Approach

Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, Philip H. S. Torr

arXiv:1812.05050v2cs.CV

TL;DR

Visual tracking is efficient but commonly represents objects with coarse bounding boxes, while video object segmentation supplies pixel-level masks but has traditionally been too slow for real-time use. SiamMask combines offline-trained Siamese tracking with multi-task bounding-box regression and binary segmentation. It runs online from one bounding-box initialization at 55 frames per second, achieves state-of-the-art real-time tracking on VOT-2018, and remains competitive while fastest by a large margin on DAVIS benchmarks.

  • Problem

    Video object segmentation offers detailed pixel-level representations but has traditionally been too slow for real-time operation, while tracking commonly uses less precise bounding boxes.

  • Method

    SiamMask jointly trains a fully-convolutional Siamese network for similarity matching, bounding-box regression, and class-agnostic binary segmentation.

  • Results

    55 frames per second: SiamMask produces masks and rotated bounding boxes from one bounding-box initialization, establishes a real-time tracking state of the art on VOT-2018, and is competitive and fastest by a large margin on DAVIS-2016 and DAVIS-2017.

  • Takeaways & Limitations

    A single simple method can perform online visual tracking and semi-supervised video object segmentation in real time without test-time adaptation.

  • Takeaways & Limitations

    SiamMask fails in motion-blur and non-object-instance scenarios associated with a lack of similar training samples.

Abstract

from arXiv · show

In this paper we illustrate how to perform both visual object tracking and semi-supervised video object segmentation, in real-time, with a single simple approach. Our method, dubbed SiamMask, improves the offline training procedure of popular fully-convolutional Siamese approaches for object tracking by augmenting their loss with a binary segmentation task. Once trained, SiamMask solely relies on a single bounding box initialisation and operates online, producing class-agnostic object segmentation masks and rotated bounding boxes at 55 frames per second. Despite its simplicity, versatility and fast speed, our strategy allows us to establish a new state of the art among real-time trackers on VOT-2018, while at the same time demonstrating competitive performance and the best speed for the semi-supervised video object segmentation task on DAVIS-2016 and DAVIS-2017. The project website is http://www.robots.ox.ac.uk/~qwang/SiamMask.

1. Introduction

Visual tracking offers efficient online localization from a bounding-box initialization, whereas video object segmentation provides more detailed masks but has traditionally lacked real-time speed. SiamMask narrows this gap by combining tracking and segmentation in one offline-trained, online method.

  • Motivation: Visual tracking estimates an arbitrary target’s position across subsequent video frames from its first-frame location.This supports applications including surveillance, navigation, video labelling, human-computer interaction, and activity recognition.
  • Motivation: Online trackers process streaming video without using future frames, typically representing targets with axis-aligned or rotated bounding boxes.Bounding-box annotation keeps labelling and target initialization quick and simple.
  • Motivation: Video object segmentation instead predicts a binary mask indicating whether each pixel belongs to the target, which is useful for pixel-level applications.Pixel-level estimates are more computationally demanding than bounding boxes.
  • Problem: VOS methods have traditionally required several seconds per frame, and even recent faster approaches have not operated in real time.This speed gap motivates a method that combines tracking convenience with segmentation detail.
  • Contribution: SiamMask uses a simple multi-task approach to address visual tracking and VOS while retaining offline training, online speed, and a more refined target representation.It augments similarity-based tracking with bounding-box regression and class-agnostic binary segmentation, requiring binary labels only during offline training.
  • Contribution: 55 frames per second: SiamMask produces segmentation masks and rotated bounding boxes after a single bounding-box initialization, operates online without updates, and achieves strong results on tracking and VOS benchmarks.It establishes a real-time tracking state of the art on VOT-2018 and is competitive and fastest by a large margin on DAVIS-2016 and DAVIS-2017.

2. Related Work

Prior work separates fast online tracking from more detailed but slower video object segmentation. SiamMask is motivated by combining offline-trained Siamese tracking with binary masks while preserving simple initialization and online operation.

  • Visual object tracking: Traditional arbitrary-object trackers commonly train a discriminative classifier online from first-frame ground-truth information and update it during tracking.
  • Visual object tracking: Fully-convolutional Siamese trackers instead learn an offline similarity function on video-frame pairs and evaluate it once per frame on new videos.Later variants improve this approach with region proposals, hard-negative mining, ensembling, and memory mechanisms.
  • Visual object tracking: Most modern trackers use rectangular bounding boxes for initialization and subsequent localization, although rectangles can poorly represent object shape.This limitation motivates producing binary masks while retaining bounding-box initialization.
  • Visual object tracking: Earlier mask-producing trackers either operated slowly, lacked modern benchmark competitiveness, or required test-time fine-tuning.Yeo et al.’s fastest variant ran at 4 frames per second, while CNN features reduced its speed below 0.1 fps.
  • Semi-supervised video object segmentation: VOS methods use temporal propagation or independent-frame processing, with examples including graph-labeling methods, optical-flow models, and first-frame fine-tuning.Recent work increasingly targets faster methods, including approaches designed to approach state-of-the-art performance.
  • Semi-supervised video object segmentation: VOS methods prioritize accurate object representation and often use computationally intensive techniques such as fine-tuning, data augmentation, or optical flow.These approaches can require minutes or hours for videos lasting only a few seconds.

3. Methodology

SiamMask extends fully-convolutional Siamese trackers with a segmentation branch, enabling pixel-wise masks alongside tracking outputs. It uses shared representations, multi-task losses, and online mask- or box-based reference selection.

  • Siamese tracking foundation: SiamMask adopts fully-convolutional Siamese tracking as its starting framework for online operation and fast speed.The method considers SiamFC and SiamRPN as representative starting points.
  • Mask prediction: The approach extends Siamese trackers with an extra branch and loss to encode pixel-wise binary masks in each response of a candidate window.It predicts one w×h mask for each candidate window using a two-layer network hφ.
  • Mask prediction: Mask prediction depends on both the search image x and exemplar target z, allowing the exemplar to guide segmentation of the search image.A different reference image produces a different segmentation mask for x.
  • Training objective: The mask loss is binary logistic regression over candidate windows, with pixel classifiers indicating whether each pixel belongs to the object.Lmask is applied only to positive candidate windows.
  • Training objective: SiamMask combines mask learning with similarity, score, and box objectives in two-branch and three-branch variants.The three-branch objective is L3B = λ1 · Lmask + λ2 · Lscore + λ3 · Lbox; training uses λ1 = 32 and λ2 = λ3 = 1.
  • Online inference: At inference, the selected mask is thresholded at 0.5, then either fitted with a Min-max box or replaced by the highest-scoring box-branch output for the next search region.The two-branch variant uses the fitted mask box, while the three-branch variant uses the box branch.

4. Experiments

Experiments evaluate SiamMask on visual tracking and semi-supervised video object segmentation, examining representations, benchmark performance, efficiency, and architectural choices. Across these settings, mask-based representations and the proposed online design deliver strong accuracy with real-time operation.

  • Evaluation settings: SiamMask is evaluated on VOT-2016 and VOT-2018 for tracking, and DAVIS-2016, DAVIS-2017, and YouTube-VOS for semi-supervised VOS.The tracking evaluation uses IOU, AP, EAO, accuracy, and robustness; VOS evaluation uses region similarity, contour accuracy, recall, mean, and decay.
  • Object representation: mAP@0.5 IOU reaches 85.4 for SiamMask-MBR, improving by +29 and +9.2 points over the two fully-convolutional baselines.At mAP@0.7 IOU, the corresponding improvements widen to +41.6 and +18.4 points.
  • Object representation: The MBR strategy for converting masks to rotated boxes offers a significant advantage over axis-aligned bounding boxes.SiamMask achieves the best mIOU across the evaluated box-generation strategies, while SiamMask-Opt has the highest IOU and mAP but requires slow optimization.
  • Tracking results: EAO reaches 0.380 at 55 frames per second for the three-branch SiamMask on VOT-2018, significantly outperforming DaSiamRPN.SiamMask-2B reaches 0.334 EAO and is superior to other real-time methods reported in the published literature.
  • VOS results: SiamMask is competitive on DAVIS-2016 and DAVIS-2017 while being four times more efficient than the fastest recent VOS methods without fine-tuning.It is almost two orders of magnitude faster than accurate approaches such as OnAVOS or SFL.
  • VOS results: SiamMask achieves very low decay for region similarity and contour accuracy, indicating stable performance over time on long sequences.The method operates online without test-sequence adaptation and uses a simple bounding-box initialization.
  • Further analysis: Replacing the shared backbone improves performance but reduces speed, while refinement is especially important for contour accuracy FM.SiamMask-2B and SiamMask improve over SiamFC and SiamRPN with the same backbone.
  • Further analysis: The two-branch and three-branch variants improve VOT-2018 EAO from 0.251 to 0.265 and from 0.359 to 0.363, respectively.The mask branch is trained but not used during inference in this ablation.

5. Conclusion

SiamMask enables fully-convolutional Siamese trackers to produce class-agnostic binary masks while addressing both visual tracking and semi-supervised video object segmentation. The method uses bounding-box initialization, operates online in real time, and requires no test-sequence adaptation.

  • The method addresses both visual object tracking and semi-supervised video object segmentation, achieving better accuracy than state-of-the-art trackers and the fastest speed among VOS methods.
  • SiamMask variants use simple bounding-box initialization, operate online, run in real time, and require no adaptation to the test sequence.

A. Architectural details

The architectural details use a ResNet-50 backbone with modified stride and dilation, lightweight network heads, and stacked refinement modules for more accurate masks.

  • Both variants use ResNet-50 through the final convolutional layer of the fourth stage as their backbone.
  • The backbone reduces output stride to 8 with stride-1 convolutions and increases receptive field using dilated convolutions.
  • The network heads use normalization and ReLU in conv5, while conv6 consists only of a 1×1 convolution.
  • Mask refinement merges low- and high-resolution features through upsampling layers and skip connections in multiple refinement modules.
  • Table 8 provides the backbone architecture and details each building block in square brackets.

B. Further qualitative results

The supplementary figures visualize SiamMask’s refinement and mask-generation process, along with qualitative outputs on tracking and video object segmentation benchmarks.

  • Tables 9 and 10 give architectural details for the three-branch and two-branch heads.
  • Figure 6 illustrates an example refinement module U3, while Figure 8 schematically shows stacked refinement modules generating masks.
  • The mask branch produces score maps at different locations and generates a mask for each RoW.
  • During inference, the score branch selects the final mask from the location with the maximum score.
  • Figures 9 and 10 provide further qualitative results on VOT-2018 and DAVIS-2016/DAVIS-2017 sequences, respectively.
Loading 1812.05050v2…