PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Object Segmentation and Tracking: A Survey

Rui Yao, Guosheng Lin, Shixiong Xia, Jiaqi Zhao, Yong Zhou

arXiv:1904.09172v3cs.CV

TL;DR

Video object segmentation and tracking face shared and task-specific difficulties that motivate their joint study. This survey organizes VOST methods, reviews their technical characteristics, and summarizes datasets and evaluation metrics. It concludes by synthesizing five method categories and discussing their advantages, disadvantages, and future directions.

  • Problem

    VOST must address shared challenges such as occlusion, deformation, motion blur, and scale variation, alongside segmentation-specific complexity and tracking-specific fast motion, out-of-view, and real-time constraints.

  • Method

    The survey hierarchically categorizes VOST into unsupervised, semi-supervised, interactive, weakly supervised, and segmentation-based tracking methods, and reviews their techniques, datasets, and metrics.

  • Results

    The survey provides a comprehensive review of VOST methods, their technical characteristics, related datasets, and pixel-wise mask and bounding-box evaluation metrics.

  • Takeaways & Limitations

    The review provides a structured reference for understanding VOST methods and their advantages or disadvantages across application scenarios.

  • Takeaways & Limitations

    CNN-based fine-tuning can fail to adapt to large appearance changes and may generalize poorly to new object appearances.

Abstract

from arXiv · show

Object segmentation and object tracking are fundamental research area in the computer vision community. These two topics are diffcult to handle some common challenges, such as occlusion, deformation, motion blur, and scale variation. The former contains heterogeneous object, interacting object, edge ambiguity, and shape complexity. And the latter suffers from difficulties in handling fast motion, out-of-view, and real-time processing. Combining the two problems of video object segmentation and tracking (VOST) can overcome their respective difficulties and improve their performance. VOST can be widely applied to many practical applications such as video summarization, high definition video compression, human computer interaction, and autonomous vehicles. This article aims to provide a comprehensive review of the state-of-the-art tracking methods, and classify these methods into different categories, and identify new trends. First, we provide a hierarchical categorization existing approaches, including unsupervised VOS, semi-supervised VOS, interactive VOS, weakly supervised VOS, and segmentation-based tracking methods. Second, we provide a detailed discussion and overview of the technical characteristics of the different methods. Third, we summarize the characteristics of the related video dataset, and provide a variety of evaluation metrics. Finally, we point out a set of interesting future works and draw our own conclusions.

1 INTRODUCTION

The survey motivates joint video object segmentation and tracking by their shared challenges and mutual benefits, then organizes prior work into five categories and reviews methods, datasets, and metrics.

  • Video object segmentation produces foreground masks, whereas tracking estimates an object's location, size, or orientation in each frame.
  • Segmentation and tracking are interdependent because accurate masks provide reliable observations for tracking, while tracking can guide segmentation.
  • Organization and contributions: It also discusses technical characteristics, video datasets, evaluation metrics, and application scenarios including video summarization, HD video compression, and human interaction.
  • Challenges and issues: VOS and VOT share challenges including occlusion, deformation, motion blur, and scale variation, while VOT additionally faces fast motion, out-of-view targets, and real-time constraints.
  • Organization and contributions: The survey groups VOST approaches into unsupervised, semi-supervised, interactive, weakly supervised, and segmentation-based tracking methods.

2 MAJOR METHODS

The survey presents major VOST method families, emphasizing unsupervised approaches based on appearance and motion cues, including background subtraction and point trajectories.

  • Unsupervised VOS: Unsupervised VOS requires no user input and commonly assumes target objects differ in motion or appear frequently across the sequence.
  • Background subtraction: Background subtraction models scene appearance and marks substantial deviations as moving-object foreground regions.
  • Background subtraction: Background-subtraction methods are categorized by stationary backgrounds, 2D parametric motion, and 3D motion.
  • Point trajectories: Point-trajectory VOS generates trajectories, clusters them using affinities, and uses the clusters as priors for segmentation.
  • Limitations: Background-subtraction approaches rely on restrictive camera assumptions and are sensitive to model selection, limiting their handling of non-rigid backgrounds.

Discussion.

Motion-based VOS methods exploit information across frames, using either dense optical flow or sparse long-term feature trajectories. Trajectory methods offer temporal consistency but can fail under occlusion, disocclusion, large motion, and non-rigid deformation.

  • Point trajectory methods: Point-trajectory VOS first generates trajectories, clusters them by affinity, and uses the clusters as priors for segmentation.The survey divides these methods by whether motion is estimated with optical flow or feature tracking.
  • Motion representations: Optical flow estimates dense inter-frame motion fields, whereas feature tracking follows sparse salient points over many frames.
  • Optical flow methods: Dense optical-flow methods commonly assume constant brightness and small displacement between adjacent frames.These assumptions support the basic optical-flow equations and estimation process.
  • Feature tracking methods: KLT-based tracking can build sparse long-term trajectories that support temporally consistent clustering across many frames.Brox and Malik’s approach clusters long-term trajectories rather than relying only on two-frame motion fields.
  • Limitations: Point trajectories may become too short under non-rigid objects or large motion because frequent occlusions and disocclusions interrupt tracking.These methods also lack object appearance information when relying only on low-level bottom-up cues.

Discussion.

Over-segmentation methods reduce video complexity by grouping pixels into space-time regions and assigning labels through graph-based models. They improve efficiency but can require additional knowledge and remain sensitive to temporal boundary changes.

  • Over-segmentation: Over-segmentation groups pixels by color, brightness, optical flow, or texture to produce spatio-temporal segmentation maps.Superpixel segmentation is a form of over-segmentation that preserves most structure needed for segmentation.
  • Graph-based models: Graph-based methods represent pixels, superpixels, or supervoxels as graph nodes and partition them into labeled subgraphs by minimizing an energy function.
  • Graph-based models: The energy function combines unary label likelihoods with weighted pairwise terms over neighboring pixels, controlled by λ.
  • Supervoxels: Supervoxel methods over-segment videos into spatio-temporal regions with uniform motion and appearance.
  • Limitations: Over-segmentation reduces computation by estimating disparities per segment rather than per pixel, but complex videos require additional knowledge and boundary stability.

Discussion.

Object-like segmentation methods move beyond low-level regions by generating foreground hypotheses from salient objects, motion proposals, or bounding boxes. These holistic proposals can improve accuracy but often introduce background false positives.

  • Object-like segments: Object-like methods generate a foreground-object hypothesis for each frame using learned models of salient regions or object proposals.
  • Proposal categories: Video object proposals commonly use figure-ground regions, optical flow, or bounding boxes as three proposal categories.
  • Figure-ground proposals: Figure-ground approaches combine cues such as color, texture, and boundaries, then group key-segments or region proposals across frames.
  • Optical-flow proposals: Optical-flow proposals estimate object displacement and generate moving-object candidates from multiple segmentations.
  • Bounding-box proposals: Bounding-box proposals combine object detectors and trackers, convert box tracks into pixel-wise segments, and refine those segment tracks.
  • Limitations: Holistic object proposals can improve segmentation accuracy through global cues but often generate background false positives that reduce performance.

Discussion.

CNN-based VOS methods replace hand-crafted representations with learned appearance and motion models, while unsupervised methods automatically initialize foreground segmentation without human annotation. Unsupervised approaches remain limited in object specificity and computational efficiency.

  • CNN transition: Earlier VOS methods relied on hand-crafted features rather than learned video representations for appearance and motion modeling.
  • CNN methods: CNN-based methods generate salient objects and propagate them through video, with later approaches using end-to-end deep convolutional networks.
  • Unsupervised VOS: Unsupervised VOS automatically segments foreground objects without human annotation by exploiting saliency, semantics, optical flow, or motion.
  • Limitations: Unsupervised methods cannot reliably select a specific object under motion confusion between instances or dynamic backgrounds.
  • Limitations: Unsupervised methods are computationally expensive because they process many unrelated object-like proposals.

2.2 Semi-supervised video object segmentation

Semi-supervised VOS starts from an initial mask and propagates object labels through subsequent frames using graph-based or CNN-based methods. Graph approaches model spatiotemporal relations over pixels, superpixels, or patches, while CNN approaches use motion or detection cues but face complementary robustness and adaptation limits.

  • Semi-supervised VOS receives an initial object mask in the first or key frames, then segments the object in the remaining frames.
  • Spatio-temporal graph methods: Graph-based methods formulate VOS as spatiotemporal label propagation over pixels, superpixels, or object patches.
  • Spatio-temporal graph methods: Pixel, superpixel, and patch graphs trade fine boundaries and computational cost while using temporal connections to infer labels across frames.
  • CNN-based methods: CNN-based methods are grouped into motion-based mask propagation and detection-based approaches according to how they use temporal information.
  • CNN-based methods: Motion-based methods can handle smooth changes but are vulnerable to occlusion, fast motion, and drift, whereas detection-based methods resist temporal discontinuities but adapt poorly to appearance changes.
  • CNN-based methods: CNN-based semi-supervised methods commonly fine-tune on the first frame, but this improves accuracy at the cost of adaptation and often more than ten minutes of test-time optimization.
  • CNN-based methods: CNN pipelines also use MRF or CRF post-processing and data augmentation to improve mask quality and generalization.

2.3 Interactive video object segmentation

Interactive VOS uses iterative user input, such as scribbles or clicks, to produce spatiotemporal object masks. Methods include graph partitioning, active contours, and CNN models, with recent approaches targeting faster and multi-object interaction.

  • Interactive VOS repeatedly incorporates user corrections through foreground or background scribbles or clicks.
  • Interactive methods are organized into graph partitioning, active contour, and convolutional neural network branches.
  • Graph partitioning models: Graph-based approaches formulate segmentation as partitioning graph vertices and include graph cuts, random walker, and geodesic methods.
  • Active contour models: Active-contour methods use image boundaries to stop curve evolution, while TouchCut reduces interaction to a single point and propagates results with optical flow.
  • CNN models: CNN-based methods improve interactive segmentation, including approaches that refine predictions with fully connected CRFs or formulate segmentation as pixel-wise retrieval.
  • Interactive VOS can produce full spatiotemporal segmentation from sparse input, but precise clicks or scribbles may be cumbersome on small touch screens.

2.4 Weakly supervised video object segmentation

Weakly supervised VOS uses videos known to contain a foreground object or class rather than dense pixel annotations. Approaches generate proposals, transfer labels, exploit unlabeled target-domain structure, or use language descriptions to identify the object.

  • Weakly supervised VOS uses large video collections in which videos are known to contain the same foreground object or object class.
  • Methods generate semantic object proposals, train segment classifiers, or transfer labels to produce target-object masks.
  • Multiple-instance learning and positive-versus-negative segment comparisons are used to identify foreground regions without dense supervision.
  • Other approaches combine CNN recognition with unlabeled target-domain structure or use natural-language expressions to identify the target object.

2.5 Segmentation-based Tracking

Segmentation-based tracking represents objects with masks or segmented regions rather than only bounding boxes, supporting deformable-object tracking. Bottom-up methods separate segmentation and tracking, while joint frameworks use feedback between them to reduce propagated errors.

  • Segmentation-based tracking uses pixel-level masks for applications such as video editing and compression, but estimating all pixels is computationally costly.
  • Bottom-up frameworks: Bottom-up methods segment regions first, match or propagate contours across frames, and are divided into contour matching and contour propagation.
  • Contour matching: Contour matching models object appearance or background distinction with generative, discriminative, or hybrid trackers and outputs the mask of the best sample.
  • Contour propagation: Contour propagation uses sequential Monte Carlo or direct minimization, with state dynamics and observation models describing motion and contour similarity.
  • Joint frameworks: Bottom-up pipelines propagate segmentation errors into tracking because foreground segmentation and tracking are performed as separate tasks.
  • Joint frameworks: Joint frameworks let segmentation guide feature extraction and tracking provide top-down cues for segmentation, exploiting their correlation.

2.5.2 Joint-based methods.

Joint-based methods unify video object segmentation and tracking within graph-based or probabilistic frameworks, typically using energy minimization to estimate object masks. CNN-based approaches increasingly address both tasks with improved speed and accuracy.

  • Graph-based methods: Graph-based joint methods construct an energy-minimization graph using graph cuts, random walkers, or shortest geodesics.These core algorithms support simultaneous tracking and segmentation.
  • Graph-based methods: Joint frameworks combine low-level image measures and high-level observations to simultaneously estimate tracked-object segmentation and motion.Examples include iterative RANSAC-style optimization and semantic extensions.
  • Probabilistic methods: Probabilistic methods jointly solve tracking and segmentation using Bayesian models, pixel-wise posteriors, or conditional random fields.These approaches connect appearance modeling, contour segmentation, and target tracking.
  • CNN-based methods: CNN methods can simultaneously estimate segmentation masks, bounding boxes, and object/background scores, sometimes from only a first-frame bounding box.Siamese and two-branch architectures combine appearance and contour information.
  • CNN-based methods: Bottom-up methods use foreground contours as features to reduce object drift in non-rigid tracking and segmentation, while focusing on target localization.Unlike traditional tracking, these methods exploit object contours rather than only appearance or bounding-box outputs.
  • Discussion: Iterative energy minimization can estimate accurate poses, motions, and occlusions but inherently limits runtime speed, whereas CNN methods can be accurate and fast.Joint methods may output masks without bounding boxes.

3 DATASETS AND METRICS

The survey reviews datasets, evaluation protocols, and metrics for video object segmentation and tracking. The benchmarks cover diverse object counts, motions, scenes, annotations, and task settings, while metrics assess masks, contours, temporal stability, and bounding-box tracking.

  • Datasets: SegTrack and SegTrack v2 evaluate tracking and video object segmentation with pixel- or instance-level annotations under deformation, similar objects, fast motion, clutter, and low contrast.SegTrack contains 6 videos, while SegTrack v2 contains 14 videos and includes sequences with multiple objects.
  • Datasets: BMS-26 and FBMS-59 are widely used for unsupervised and semi-supervised video object segmentation.FBMS-59 extends the earlier BMS-26 collection with additional sequences.
  • Datasets: JumpCut is considered challenging because its sequences contain very fast object motion and significant foreground deformation.It contains 22 sequences across 14 categories with 6,331 annotated images.
  • Datasets: DAVIS, NR, MOT, VOT, and OTB support evaluation across single-object, multiple-object, segmentation, and tracking settings.Their annotations include pixel-level masks, instance labels, or bounding boxes depending on the dataset.
  • Metrics: VOS evaluation measures region similarity, contour precision, and temporal stability, while tracking evaluation uses bounding-box and location metrics.Region similarity is based on mask intersection over union; OTB uses VOR, CLE, DP, and OP under one-pass evaluation.
  • Metrics: OTB 2013 and VOT 2016 provide widely used protocols for evaluating single-object tracking algorithms.OTB 2013 evaluates bounding-box overlap, center location error, distance precision, and overlap precision.

4 FUTURE DIRECTIONS

The survey identifies simultaneous VOS and VOT, fine-grained video analysis, and generalization as important future directions. These directions reflect unresolved trade-offs between accuracy, speed, detail, and transfer to new settings.

  • Simultaneous prediction: Simultaneous prediction of masks and bounding boxes remains constrained by the trade-off between accuracy and processing speed.Fine-tuning iterations can improve segmentation and tracking performance but are computationally expensive.
  • Fine-grained VOST: Fine-grained objects in full-HD video are difficult to segment and track because small object parts can be obscured by varied backgrounds and motion.Those parts may contain important semantic information for fine-grained classification.
  • Generalization: Generalization to new categories and unconstrained settings remains a difficulty even after VOST tasks are solved through training.The survey presents transfer beyond trained categories as an open direction.

5 CONCLUSION

The paper provides a comprehensive survey of video object segmentation and tracking, organizing methods into five categories and reviewing their technical characteristics. It also covers datasets, evaluation metrics, method trade-offs, and future research directions.

  • Survey scope: The survey classifies VOST literature into unsupervised, semi-supervised, interactive, weakly supervised, and segmentation-based tracking methods.It uses a hierarchical categorization to organize existing approaches.
  • Technical synthesis: The review summarizes object representations, image features, motion cues, CNN-based processing stages, and methodological advantages and disadvantages.This analysis spans different algorithmic approaches in the VOST literature.
  • Evaluation resources: The survey describes related video datasets and evaluation metrics for pixel-wise masks and bounding boxes.These resources support evaluation of both segmentation and tracking techniques.
  • Implications: The authors present the review as a source of insights intended to support researchers and encourage future VOST work.This conclusion states the survey's intended research utility without asserting a measured outcome.
Loading 1904.09172v3…