PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Object Segmentation and Tracking: A Survey

Rui Yao, Guosheng Lin, Shixiong Xia, Jiaqi Zhao, Yong Zhou

arXiv:1904.09172v3cs.CV

TL;DR

Video object segmentation and tracking face shared and task-specific challenges, including occlusion, motion blur, fast motion, and shape complexity. This survey hierarchically classifies VOST methods and reviews their characteristics, datasets, and metrics, while discussing method advantages and disadvantages.

  • Problem

    Video object segmentation and tracking face common and task-specific challenges, including occlusion, motion blur, fast motion, and shape complexity.

    Source: §1.1 · p. 3

  • Method

    The paper surveys VOST literature by hierarchically classifying five method categories and summarizing method characteristics, related datasets, and evaluation metrics.

    Source: §5 · pp. 30–31

  • Results

    The review discusses advantages and disadvantages across VOST methods and describes pixel-wise mask and bounding-box evaluation metrics.

    Source: §5 · p. 31

  • Takeaways & Limitations

    The authors expect the review to provide useful insights into VOST and hope it encourages further research.

    Source: §5 · p. 31

    Fine-tuned CNN-based semi-supervised VOS methods may not adapt to large appearance changes and may generalize poorly to new object appearances.

    Source: §2.2.2.3 · p. 17

Abstract

from arXiv · show

Object segmentation and object tracking are fundamental research area in the computer vision community. These two topics are diffcult to handle some common challenges, such as occlusion, deformation, motion blur, and scale variation. The former contains heterogeneous object, interacting object, edge ambiguity, and shape complexity. And the latter suffers from difficulties in handling fast motion, out-of-view, and real-time processing. Combining the two problems of video object segmentation and tracking (VOST) can overcome their respective difficulties and improve their performance. VOST can be widely applied to many practical applications such as video summarization, high definition video compression, human computer interaction, and autonomous vehicles. This article aims to provide a comprehensive review of the state-of-the-art tracking methods, and classify these methods into different categories, and identify new trends. First, we provide a hierarchical categorization existing approaches, including unsupervised VOS, semi-supervised VOS, interactive VOS, weakly supervised VOS, and segmentation-based tracking methods. Second, we provide a detailed discussion and overview of the technical characteristics of the different methods. Third, we summarize the characteristics of the related video dataset, and provide a variety of evaluation metrics. Finally, we point out a set of interesting future works and draw our own conclusions.

1 INTRODUCTION

The survey reviews video object segmentation and tracking (VOST) as complementary tasks whose integration can improve both. It organizes their methods and examines challenges, datasets, evaluation metrics, and future research.

1 INTRODUCTION

  • Accurate segmentation supplies tracking with reliable object observations, while tracking guides segmentation toward object positions, helping address their respective difficulties.The survey presents VOS and VOT as interrelated tasks and describes segmentation-based tracking as combining them.Source: §1 · pp. 2–3
  • VOS methods produce precise masks but can be slow, whereas VOT methods process long sequences quickly but often output boxes or ellipses that can drift on non-rigid objects.Segmentation-based tracking combines bottom-up and top-down approaches to efficiently propagate contours of deformable or articulated objects.Source: §1 · pp. 2–3
  • VOST has applications in video summarization, video compression, retrieval, web browsing, and gesture control.The survey also describes its use in MPEG-4 content-based coding and in estimating non-rigid targets for tracking and mask description.Source: §1 · p. 3

1.1 Challenges and issues

  • VOS and VOT share challenges including occlusion, deformation, motion blur, and scale variation; tracking also faces fast motion and out-of-view objects, while segmentation faces edge ambiguity and shape complexity.The survey also identifies background clutter, low resolution, heterogeneous objects, interacting objects, and real-time processing as challenges.Source: §1.1 · p. 3

1.2 Organization and contributions of this survey

  • The survey groups VOST approaches into five categories: unsupervised, semi-supervised, interactive, weakly supervised VOS, and segmentation-based tracking.It further categorizes methods within each group and frames the review around application scenarios, object representations, features, motion modeling, CNN processing, and evaluation.Source: §1.2 · pp. 3, 5, §1.1 · p. 3
  • The survey reviews methods’ technical characteristics, summarizes video datasets and evaluation metrics for masks and bounding boxes, and identifies future research issues.Its coverage includes VOS and segmentation-based tracking approaches, along with image features, object representations, and CNN pre- and post-processing.Source: §1.2 · pp. 4–5

2 MAJOR METHODS

The survey divides VOST methods into five categories: unsupervised, semi-supervised, interactive, weakly supervised, and segmentation-based tracking. This section focuses on unsupervised VOS and background subtraction, including stationary, 2D-motion, and 3D-motion backgrounds.

2.1 Unsupervised video object segmentation

  • Unsupervised VOS requires no user input and finds objects automatically, generally assuming they move differently or appear frequently across frames.The survey groups unsupervised methods into five groups.Source: §2.1 · p. 5

2.1.1 Background subtraction.

  • Background subtraction builds a scene background model and marks pixels that deviate from it as foreground, with methods categorized by stationary, 2D, or 3D background motion.The classic process detects changed pixels and uses connected components to estimate object regions; early geometric methods were limited to specific motion backgrounds.Source: §2.1.1 · p. 5

2.1.1.1 Stationary backgrounds.

  • Stationary-background modeling ranges from Wren et al.’s single-Gaussian pixel model to multimodal models, including Stauffer and Grimson’s MoG, KDE, and combined KDA/SVM classification.Gao et al. found a single Gaussian insufficient to account for acquisition noise. The figure illustrates segmentation based on point-trajectory clustering.Source: §2.1.1.1 · pp. 5–6, Figure 2 · p. 6

2.1.1.2 Backgrounds undergoing 2D parametric motion.

  • For 2D parametric motion, methods use temporal integration to preserve tracked-object sharpness, spatial Gaussian distributions, cue fusion and priors, or sample-based background subtraction.Source: §2.1.1.2 · p. 6
  • Irani and Anandan describe 2D processing for planar scenes with camera rotation or scaling and 3D processing for significant depth change with camera translation; Torr and Zisserman use rigid-motion constraints.Brutzer et al. evaluate background-subtraction methods, identify challenges, and compare methods with post-processing.Source: §2.1.1.3 · p. 6

2.1.1.3 Backgrounding undergoing 3D motions.

  • Background-subtraction methods rely on restrictive assumptions about stable or slowly moving cameras, are sensitive to 2D-versus-3D model selection, and cannot handle special backgrounds such as non-rigid objects.Source: Discussion · p. 6

Discussion.

The section presents long-range motion and point trajectories as a basis for video object segmentation: trajectories are clustered by affinity and used as segmentation priors. It distinguishes dense optical flow from sparse feature tracking and notes limitations of trajectory-based methods.

Discussion.

  • Long-term motion supports video object segmentation by clustering point trajectories through their affinity matrix and using the clusters as segmentation priors.The survey describes motion as a strong perceptual cue and notes that these methods use information across multiple frames.Source: §2.1.2 · p. 6

2.1.2 Point trajectory.

  • Point-trajectory VOS methods use either dense optical flow between adjacent frames or sparse feature tracking of salient points over many frames.Optical flow estimates a dense motion field, while feature tracking follows sparse points.Source: §2.1.2 · p. 7

2.1.2.1 Optical flow based methods.

  • Basic optical flow assumes constant brightness and small displacement between adjacent frames.The small-movement assumption means object position does not change drastically between frames.Source: §2.1.2.1 · p. 7
  • Optical-flow trajectory methods address difficult flow discontinuities and sparse tracks with clustering, dense segmentation, and motion-grouping approaches.Flow estimation is difficult in low-texture or low-motion regions, where smoothing constraints can interpolate flow but require segmentation to preserve motion discontinuities. Examples include motion profiles with normalized cuts, long-range coherent track clusters, dense segmentation from sparse clusters, context-aware saliency maps, and probabilistic 3D segmentation combining spatial, temporal, and semantic information.Source: §2.1.2.2 · p. 7, §2.1.2.1 · p. 7

2.1.2.2 Feature tracking based methods.

  • Trajectory methods can capture objects over long time spans, but non-rigid objects and large motion cause occlusions that shorten tracks, and the methods lack object appearance information.Unlike short-term background subtraction, trajectory methods can fail when objects are static in some frames.Source: Discussion · p. 7

Discussion.

The section surveys video segmentation methods built on over-segmentation, describing their region representations and tradeoffs. These methods reduce computation, but complex videos can require additional knowledge and remain sensitive to boundary changes.

2.1.3 Over-segmentation.

  • Over-segmentation groups pixels by color, brightness, optical flow, or texture into space-time regions, but growing segment counts can make graph-based optimization intractable.Graph methods partition pixel, superpixel, or supervoxel nodes by pruning weighted edges and minimizing unary label and pairwise-neighbor costs.Source: §2.1.3 · p. 8

2.1.3.1 Superpixel representation.

  • Chang et al. quantify temporal superpixel consistency, Giordano et al. refine coarse foregrounds, and Yang et al. analyze multiple labeling granularities.Superpixel segmentation is an over-segmentation that preserves most structure needed for segmentation.Source: §2.1.3.1 · pp. 8, 10, §2.1.3 · p. 8
  • Supervoxels extend superpixels into video, grouping space-time regions by uniform motion and appearance; methods use hierarchical graphs or randomized merging.Grundmann et al. propose hierarchical graph-based segmentation of long videos, while Oneata et al. generate proposals through supervoxel merging.Source: §2.1.3.2 · p. 10

2.1.3.2 Supervoxel representation.

  • Other methods use boundaries or patches; over-segmentation lowers per-segment computation, but complex videos require extra knowledge and boundary fluctuations cause sensitivity.Wang uses a modified watershed transformation, Stein et al. use motion cues for object or occlusion boundaries, and patch-based methods include a hypergraph cut and a coupled graphical model.Source: §2.1.3.2 · pp. 10–11, Discussion · p. 11

Discussion.

The discussion describes object-like regions as a way to move from low-level pixel grouping toward frame-level foreground hypotheses. It reviews saliency and object-proposal methods, noting their potential to capture whole objects as well as their false-positive and computational costs.

2.1.4 “Object-like” segments.

  • Object-like segments use learned object-region models to generate foreground hypotheses for each frame, upgrading pixel, superpixel, or supervoxel grouping.Examples of object-like regions include salient objects and proposals generated from background information.Source: §2.1.4 · p. 11

2.1.4.1 Salient objects.

  • Saliency-based methods use visual importance as prior knowledge to identify objects, combining cues such as motion, appearance, edges, and temporal consistency.Approaches include saliency maps with CRF or graph-cut inference and consensus voting across recurring regions.Source: §2.1.4.1 · p. 11

2.1.4.2 Object proposals.

  • Object proposals are ranked to build object and background models, typically using figure-ground regions, optical-flow proposals, or bounding-box proposals.Static proposals draw on cues such as color, texture, and boundaries, then cluster or group regions across frames; some methods refine segment tracks or handle complete occlusions.Source: §2.1.4.2 · pp. 11–12
  • Optical-flow proposals select moving objects from flow boundaries or motion scores, while bounding-box methods track boxes and convert them into pixel-wise segments.Motion-based grouping can also use occlusion boundaries and long-term occlusion relations to distinguish coherent objects from internal regions.Source: §2.1.4.2 · p. 12
  • Holistic proposals can capture whole objects using motion, boundaries, semantics, shape, and global appearance, but false positives and heavy computation can reduce performance.Generating and associating thousands of segments is computationally demanding.Source: Discussion · p. 12

Discussion.

The discussion contrasts earlier hand-crafted appearance and motion models with CNN-based video segmentation, including unsupervised methods that propagate automatically generated foregrounds but face instance-confusion and computational limits.

Discussion.

  • Before deep CNNs, video-object segmentation methods relied on hand-crafted features rather than learned video representations for appearance and motion models.Source: §2.1.5 · p. 12

2.1.5 Convolutional neural networks (CNN).

  • CNN methods use dual-branch networks for video segmentation: MP-Net predicts motion labels, Tokmakov et al. [160] add appearance and GRU visual memory, FSEG uses mined data, and SfM-Net [166] learns without mask annotations.SfM-Net combines motion and structure streams to learn object masks and motion models through differentiable rendering.Source: §2.1.5 · p. 13
  • Li et al. adapt image-based instance embeddings to video segmentation and propagate pixel labels with a motion-based bilateral network and graph cut; a deep reinforcement-learning method [57] detects moving objects for action selection.Source: §2.1.5 · p. 13
  • Song et al. apply pyramid-dilated bidirectional ConvLSTM video salient-object detection to unsupervised VOS, while Wang et al. propose visual-attention-driven unsupervised VOS and collect human-attention data from DAVIS, Youtube-Objects, and SegTrack v2.Source: §2.1.5 · p. 13
  • Unsupervised VOS automatically segments foreground objects in the initial frame and propagates them using cues such as saliency, semantics, optical flow, or motion, without annotation or user interaction.It cannot reliably select a specific instance amid motion confusion or dynamic backgrounds, and unrelated object-like proposals make it computationally expensive.Source: §2.1.6 · p. 13

2.2 Semi-supervised video object segmentation

Semi-supervised VOS starts with an object mask in the first frame or key frames, then segments the object in subsequent frames. The survey groups approaches into spatio-temporal graph and CNN-based methods.

2.2.1 Spatio-temporal graphs.

  • Graph-based methods formulate segmentation as spatiotemporal label propagation over pixels, superpixels, or object patches to infer labels in later frames.Pixel graphs preserve fine boundaries; superpixel graphs reduce the computational cost and noisy links of pixel-based graphs, while patch methods seek temporally coherent segments.Source: §2.2.1.1 · pp. 13–14

2.2.1.2 Spatio-temporal connections.

  • Graph methods estimate temporal links with lattices, nearest-neighbor fields, trees, or appearance, then choose local adjacent-frame inference or global all-frame optimization.Local strategies can segment on-the-fly for sequential data; global solutions address the limits of short-range interactions.Source: §2.2.1.2 · p. 15

2.2.2 Convolutional neural networks.

  • CNN-based semi-supervised VOS methods are grouped into motion-based and detection-based approaches according to whether they use temporal motion information.The survey describes temporal motion as an important video-segmentation cue.Source: §2.2.2 · p. 15, §2.2.2.3 · p. 16

2.2.2.1 Motion-based methods.

  • Motion-based CNN methods propagate masks from an annotated frame, using optical flow or recurrent networks to model temporal coherence.Optical-flow systems may combine color-segmentation and flow branches, while MaskRNN fuses binary segmentation and localization outputs with flow.Source: §2.2.2.1 · p. 15

2.2.2.2 Detection-based methods.

  • Detection-based methods identify the object through per-frame appearance models or pixel-level matching, rather than relying on temporal propagation.Some appearance models are fine-tuned on the first-frame annotation; matching methods compare pixels or retrieve features in an embedding space.Source: §2.2.2.2 · p. 16

2.2.2.3 Discussion.

  • Motion propagation can fail under occlusion or fast motion and drift, whereas detection is more robust to these disruptions but struggles with appearance changes and similar instances.Fine-tuning the first-frame model can improve accuracy, but may hinder adaptation to major appearance changes; some methods update online.Source: §2.2.2.3 · p. 17
  • CNN-based methods often require costly test-time fine-tuning, while some alternatives avoid it; post-processing and data augmentation are additional techniques discussed.The survey reports that fine-tuning usually takes more than ten minutes to update a model, and describes contour refinement and augmented first-frame training examples.Source: §2.2.2.3 · pp. 17–18

2.3 Interactive video object segmentation

Interactive video object segmentation uses iterative user input, such as scribbles or clicks, to correct masks and produce full spatio-temporal segmentations; its methods include graph partitioning, active contours, and CNNs.

2.3.1 Graph partitioning models.

  • Graph-partitioning methods model segmentation as dividing graph vertices into disjoint subgraphs, using approaches such as graph cuts, random walker, and geodesic methods.GrabCut-based work alternates appearance-model estimation with graph-cut refinement, while other methods use superpixels or region proposals.Source: §2.3.1 · p. 19, §2.3.1.1 · p. 19

2.3.1.1 Graph-cuts based.

  • GrabCut-based approaches iteratively refine masks with graph cuts; other methods reduce min-cut computation, propagate masks across frames, or adapt local classifiers [140, 15, 172, 104, 132, 8, 7].Some approaches use optical flow to transform neighborhood regions when retraining classifiers, while others use superpixels or boundary clicks to fit region proposals.Source: §2.3.1.1 · p. 19

2.3.1.2 Random walker based.

  • A stroke-based approach combines optical flow and point trajectories, letting users draw first-frame scribbles before running the random walker.Source: §2.3.1.2 · p. 19

2.3.1.3 Geodesic based.

  • Geodesic methods use region color and pixel-value distributions to form selections, while another method combines geodesic dynamic models with pyramid-histogram confidence maps.Source: §2.3.1.3 · p. 19

2.3.2 Active contours models.

  • TouchCut simplifies active-contour interaction to one touch in the first frame, then propagates the segmentation with optical flow.The active-contour framework uses an edge detector to halt curve evolution at the target boundary.Source: §2.3.2 · p. 19

2.3.3 CNN models.

  • CNN-based methods improve interactive segmentation and can handle multiple objects; examples use CRF refinement, pixel-wise retrieval, or positive and negative point inputs [13] [23] [29].Interactive methods aim to reduce annotation time, although precise clicks or scribbles can be cumbersome on small touch screens; comparisons consider object count, input type, and pixel versus superpixel over-segmentation.Source: §2.3.3 · pp. 19–20, §2.3.4 · p. 20, Table 4 · p. 20

2.4 Weakly supervised video object segmentation

Weakly supervised VOS exploits videos known to contain the same foreground object or class, using proposals, label transfer, or other signals to identify target objects. Approaches range from training-based classifiers to methods that avoid training, proposals, or video-level annotations.

2.4 Weakly supervised video object segmentation

  • Videos known to contain the same foreground object or class support weakly supervised VOS, which generates semantic object proposals to train segmentation classifiers.Some approaches also use label transfer to produce target objects in videos.Source: §2.4 · p. 20
  • Hartmann et al. use multiple-instance learning on spatio-temporal segments, while Tang et al. distinguish foreground by appearance and Liu et al. extend classification to multiple classes.These training-example-dependent methods may produce inaccurate segmentation results.Source: §2.4 · p. 21
  • Zhang et al. use object detection without training, while Tsai et al. link objects across videos and optimize a graph without object proposals or video-level annotations.Source: §2.4 · p. 21
  • Wang et al. combine CNN recognition and representation with unlabeled target-domain structure to improve inference; Khoreva et al. [84] use natural-language expressions to identify video targets.Khoreva et al. incorporate textual descriptions of interest as foreground into convnet-based techniques.Source: §2.4 · p. 21

2.5 Segmentation-based Tracking · 2.5.1 Botom-up based methods.

Segmentation-based tracking combines pixel-level shape descriptions with tracking to better handle deformation, while bottom-up methods segment frame regions and match or propagate them across video. The review distinguishes contour matching from contour propagation and notes that separating segmentation from tracking allows segmentation errors to carry forward.

2.5 Segmentation-based Tracking

  • Segmentation provides pixel-level object shape, while box-based tracking is efficient but struggles with large deformations, motivating methods that integrate segmentation into tracking.Segmentation masks are useful when applications need pixel-level information, although estimating every pixel is computationally costly.Source: §2.5 · p. 21

2.5.1 Botom-up based methods.

  • Bottom-up methods segment regions in each frame and match or propagate them across video, using contour matching to find current-frame regions or contour propagation to move an initial contour.These methods represent objects by segmented areas rather than bounding boxes, a natural approach for non-rigid and deformable objects.Source: §2.5.1 · p. 21

2.5.1.1 Contour matching.

  • Generative contour matching selects current-frame regions using appearance or shape models, including fragment histograms, ICA, shifted subspaces, distance measures, spatiotemporal filters, and spectral matching.Examples include adaptive appearance-based fragments, trajectory-informed growing seeds, and superpixel graphs matched through spectral clustering.Source: §2.5.1.1 · p. 22
  • Discriminative contour matching classifies local patches or regions as foreground or background, with boosting and Hough-based models producing masks from selected samples.Graph-cut segmentation can be slow and increase drift risk when regions are segmented incorrectly; Duffner and Garcia propose a fast generalized-Hough detector and update the global segmentation model.Source: §2.5.1.1 · pp. 22–23
  • Hybrid generative-discriminative tracking fuses template, Hough-voting, and feature-histogram mean-shift trackers, with the fusion also providing segmentation.Source: §2.5.1.1 · p. 23

2.5.1.2 Contour propagation.

  • Contour propagation uses particle-filter methods to estimate contour shape and motion states from observations, with recursive motion and observation models guiding state updates.Wang et al. use superpixels and observation confidence; Belagiannis et al. use segmentation-based sampling for deformation, occlusion, orientation, scale, and appearance changes.Source: §2.5.1.2 · pp. 23–24
  • Direct minimization methods use gradient descent; Gu et al. combine supervised segmentation via mathematical morphology with unsupervised tracking via partial derivatives.Source: §2.5.1.2 · p. 24
  • Because segmentation errors propagate into tracking in bottom-up methods, joint frameworks use tracking cues to guide segmentation and categorize approaches as graph-based, probabilistic, or CNN frameworks.The review states that exploiting the correlation between the tasks improves video segmentation and tracking performance.Source: §2.5.2 · p. 24

2.5.2 Joint-based methods. · 2.5.3 Discussion.

Joint-based VOST methods combine segmentation and tracking in graph-based or probabilistic frameworks, using energy minimization to estimate object masks. Their iterative optimization can limit runtime, while CNN-based approaches have recently demonstrated accurate and fast joint processing.

2.5.2 Joint-based methods.

  • Joint-based methods construct a graph for energy minimization, using core algorithms such as graph cuts, random walker, and shortest geodesics.Source: §2.5.2.1 · p. 24

2.5.2.1 Graph-based framework.

  • Bugeau and Pérez jointly minimize pixelwise and high-level costs, while Wen et al., Yao et al., and Keuper et al. combine tracking and segmentation in graph-based frameworks.The approaches include iterative RANSAC-style optimization, semantic localization, and integration of bottom-up motion segmentation with top-down multiple-object tracking.Source: §2.5.2.1 · pp. 24–25
  • Bayesian approaches jointly estimate tracking and segmentation using pixel-level evidence, joint object-appearance likelihoods, image likelihoods, or superpixel-based state propagation.Aeschliman et al. update a probability model with pixel-level segmentation; Zhao et al. use Markov chain Monte Carlo, and Yeo et al. propagate segmentation online.Source: §2.5.2.2 · p. 25

2.5.2.2 Probabilistic framework.

  • Pixel-wise posterior methods build probabilistic appearance models, perform level-set segmentation, and propagate contours by gradient descent.Tjaden et al. segment multiple 3D objects and track pose with pixel-wise second-order optimization; CRF methods jointly estimate object masks, poses, or regions.Source: §2.5.2.2 · pp. 25–26

2.5.2.3 CNN framework.

  • CNN-based methods jointly process tracking and semi-supervised segmentation: Wang et al. [174] estimate masks, boxes, and object/background scores, while Zhang et al. mutually refine tracking and segmentation outputs.Zhang et al. use appearance and contour branches after receiving only a first-frame bounding box; bottom-up methods instead estimate a bounding box or coarse mask and run faster than joint-based methods.Source: §2.5.2.3 · p. 26, §2.5.3 · p. 26

2.5.3 Discussion.

  • Joint graph-based and probabilistic methods estimate exact object masks through repeated energy minimization, which can limit runtime and may yield masks without bounding boxes.The iterations estimate object poses, motions, and occlusions.Source: §2.5.3 · p. 26
  • Recent offline and online CNN-based methods demonstrate accurate and very fast simultaneous tracking and segmentation.Source: §2.5.3 · p. 26

3 DATASETS AND METRICS

The section surveys datasets used to evaluate video object segmentation and tracking, then distinguishes pixel-wise segmentation metrics from bounding-box tracking protocols. The datasets span varied object counts, scene conditions, and tracking tasks.

3.1 Video object segmentation and tracking datasets

  • SegTrack v2 moving-object annotations are provided in all frames; unlike SegTrack, many SegTrack v2 videos contain multiple objects, and some include cluttered or dynamic scenes and low-contrast object boundaries.Source: §3.1 · p. 27
  • BMS-26 and FBMS-59 support unsupervised and semi-supervised VOS, while JumpCut challenges these methods with fast motion and foreground deformation.BMS-26 has 26 videos and 189 annotated frames; FBMS-59 has 59 sequences split into 29 training and 30 test videos, and adds unconstrained-video challenges. JumpCut contains 22 sequences and 6,331 annotated images.Source: §3.1 · pp. 27–28
  • YouTube-objects and YouTube-VOS provide Internet-video datasets, while DAVIS evolves from single-object segmentation to multiple objects and interactive evaluation.YouTube-VOS contains 3,252 clips and 133,886 object annotations; DAVIS 2017 adds multiple-object labels and attributes such as occlusion and fast motion, and DAVIS 2018 adds 100 videos and an interactive teaser track.Source: §3.1 · pp. 27–28
  • NR, MOT 2016, VOT 2016, and OTB datasets address segmentation-based or bounding-box tracking across non-rigid, crowded, single-object, and real-world scenarios.NR has 11 sequences with pixel and bounding-box annotations; MOT 2016 has 14 crowded sequences, VOT 2016 has 60 sequences, and OTB 2015 has 100 sequences and 58,000 annotated frames.Source: §3.1 · p. 28

3.2.1 Evaluating of pixel-wise object segmentation techniques.

  • Pixel-wise segmentation evaluation uses region similarity J, contour precision F, and temporal stability T to assess overlap, boundary quality, and contour consistency across frames.J is based on intersection over union; F uses contour precision and recall, while T measures changes in contour shape-context descriptors between adjacent frames.Source: §3.2.1 · pp. 28–29

3.2.2 Evaluating of bounding box based object tracking techniques.

  • Single-object tracking protocols use OTB’s VOR, CLE, DP, and OP or VOT’s accuracy, robustness, and EAO; multi-target metrics include MOTA, MOTP, and trajectory completeness.VOT robustness counts target losses, while EAO estimates bounding-box accuracy after a specified number of frames; MOTA combines tracking errors and MOTP measures object localization.Source: §3.2.2 · p. 29

4 FUTURE DIRECTIONS

The section identifies five future directions for VOST: balancing speed and accuracy, handling fine-grained objects, improving generalization, and extending methods to multi-camera and 3D settings. These directions address practical challenges in segmentation and tracking across varied videos and applications.

Simultaneous prediction of VOS and VOT

  • End-to-end methods that jointly predict pixel-level masks and object-level bounding boxes must balance accuracy with processing speed.The authors identify faster processing without loss of segmentation and tracking performance as an important direction, while noting that accuracy-improving fine-tuning can be computationally expensive.Source: §4 · p. 29

Fine-grained video object segmentation and tracking

  • Fine-grained VOST must accurately track small object parts in full HD video, which can carry semantic information important for fine-grained classification.Large, varied backgrounds make these parts difficult to segment and track; distinguishing small motion differences also matters for tracking and video recognition.Source: §4 · p. 30

Generalization performance of VOST

  • VOST should generalize to new categories and unconstrained videos, since existing datasets differ from real environments and appearance and motion diversity affect performance.The authors point to noisy, compressed, unstructured, multi-view videos and call for fast and accurate segmentation and tracking in new settings.Source: §4 · p. 30

Multi-camera video object segmentation and tracking

  • Multi-camera VOST can fuse viewpoints to improve analysis of a shared scene, but methods must account for substantial visual perspective differences across camera locations.Source: §4 · p. 30

3D video object segmentation and tracking

  • 3D VOST can support collision-avoiding path planning and robotic object handling, and automate cutting objects of interest from detailed 3D building reconstructions.The authors describe segmentation with 3D images as providing detailed object boundaries, while building models can contain semantic regions for video statistics.Source: §4 · p. 30

5 CONCLUSION

The article surveys video object segmentation and tracking (VOST), organizing its literature into five categories and reviewing methods, datasets, and evaluation metrics. It aims to provide researchers with useful insights and encourage future work.

5 CONCLUSION

  • The survey classifies VOST literature into unsupervised, semi-supervised, interactive, and weakly supervised VOS, plus segmentation-based tracking.It also describes field challenges and potential applications.Source: §5 · pp. 30–31
  • It analyzes object representations, image features, motion cues, and per-process and post-process CNN-based VOS methods, discussing their advantages and disadvantages.Source: §5 · p. 31
  • The review describes related video datasets and evaluation metrics for pixel-wise mask and bounding-box-based techniques.Source: §5 · p. 31
  • The authors hope the review provides useful insights for researchers and encourages further work on VOST.Source: §5 · p. 31

Related papers

Loading 1904.09172v3…