PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning Technique for Video Segmentation

Tianfei Zhou, Fatih Porikli, David Crandall, Luc Van Gool, Wenguan Wang

arXiv:2107.01153v4cs.CV

TL;DR

Video segmentation is important across many practical applications, but existing surveys were often outdated or narrow. This survey synthesizes two major branches, reviews methods and datasets, benchmarks representative systems, and identifies open challenges; it reports notable progress while highlighting persistent evaluation and dataset limitations.

  • Problem

    Video segmentation supports many applications, while existing surveys often predate modern deep learning or cover only narrow foreground/background settings.

  • Method

    The survey reviews video object and semantic segmentation across task settings, methods, datasets, benchmarks, terminology, and future research directions.

  • Results

    The survey summarizes more than 150 deep-learning models, 20 widely used datasets, and benchmark results on 7 widely used datasets.

  • Takeaways & Limitations

    Video segmentation has achieved notable progress through deep learning, but several challenges remain for future research.

  • Takeaways & Limitations

    Many methods omit execution-time and memory reporting, while several established datasets have nearly reached performance saturation.

Abstract

from arXiv · show

Video segmentation -- partitioning video frames into multiple segments or objects -- plays a critical role in a broad range of practical applications, from enhancing visual effects in movie, to understanding scenes in autonomous driving, to creating virtual background in video conferencing. Recently, with the renaissance of connectionism in computer vision, there has been an influx of deep learning based approaches for video segmentation that have delivered compelling performance. In this survey, we comprehensively review two basic lines of research -- generic object segmentation (of unknown categories) in videos, and video semantic segmentation -- by introducing their respective task settings, background concepts, perceived need, development history, and main challenges. We also offer a detailed overview of representative literature on both methods and datasets. We further benchmark the reviewed methods on several well-known datasets. Finally, we point out open issues in this field, and suggest opportunities for further research. We also provide a public website to continuously track developments in this fast advancing field: https://github.com/tfzhou/VS-Survey.

1 INTRODUCTION

Video segmentation is a challenging computer-vision problem with applications across analysis, editing, robotics, driving, surveillance, and conferencing. This survey organizes the modern deep-learning literature across two major branches, eight sub-fields, algorithms, datasets, benchmarks, and future directions.

  • Video segmentation identifies key objects with specified properties or semantics and supports applications including autonomous driving, robotics, surveillance, augmented reality, and video conferencing.
  • Traditional approaches use hand-crafted features, heuristic priors, low- and mid-level representations, and classical machine-learning models.
  • Existing surveys often predate modern deep learning or focus narrowly on foreground/background segmentation.
  • It reviews task formulations, taxonomies, algorithms, datasets, performance evaluations, open questions, and future research directions.
  • The survey focuses on video object segmentation and video semantic segmentation, further divided into eight sub-fields.

2 BACKGROUND

The background defines video-segmentation categories, inference modes, and learning paradigms, while tracing the field from traditional methods toward deep learning. It also distinguishes related tasks and clarifies terminology used across settings.

  • Learning Paradigms for Video Segmentation: Deep-learning models use supervised, unsupervised, or weakly supervised learning strategies to approximate the video-to-segment mapping.
  • Video Segmentation Category: Video segmentation divides into VOS, which separates dominant unknown-category objects from background, and VSS, which parses predefined semantic categories.
  • Inference Modes for Video Segmentation: VOS inference modes are automatic, semi-automatic, and interactive, distinguished by the amount of human intervention during inference.
  • Terminology: The survey recommends “automatic” and “semi-automatic” instead of ambiguous VOS terms such as “unsupervised” and “semi-supervised” for inference settings.
  • History and Terminology: Earlier video methods included over-segmentation, background subtraction, motion segmentation, object proposals, tracking, and human-initialized propagation.
  • History and Terminology: Traditional approaches were constrained by hand-crafted features and heavy engineering, whereas deep learning raised video-segmentation performance.

3 DEEP LEARNING-BASED VIDEO SEGMENTATION

The survey organizes deep learning-based video segmentation into video object segmentation and video semantic segmentation, covering task settings and representative method families. It reviews automatic, semi-automatic, interactive, and language-guided object segmentation alongside methods that exploit temporal information or annotation-efficient learning for semantic segmentation.

  • Video Object Segmentation: Video object segmentation extracts generic foreground objects without recognizing semantic categories and is divided by human intervention into automatic, semi-automatic, and interactive settings.Language-guided video object segmentation is reviewed separately because it uses a specific multimodal setup.
  • Automatic Video Object Segmentation: Automatic video object segmentation learns generic object patterns from data, progressing from deep modules and pixel-level estimates toward stronger instance embeddings and cross-frame context.Representative models use frame-pair correlations, reference frames, and temporal aggregation to locate objects.
  • Semi-automatic Video Object Segmentation: Semi-automatic methods mainly propagate first-frame masks, using online fine-tuning, mask propagation, template matching, optical-flow alignment, refinement, re-identification, or external memory.External memory supports learning object evolution and using past segmentation cues over long periods.
  • Interactive Video Object Segmentation: Interactive methods reduce annotation demands through scribble-based batch segmentation or repeated interaction-propagation rounds that trade user time against segmentation quality.The survey notes that detailed first-frame masks are tedious to acquire, motivating interactive alternatives.
  • Language-guided Video Object Segmentation: Language-guided video object segmentation aligns visual and linguistic modalities, including dynamic convolution methods whose filters are generated from linguistic queries.The survey reports that linguistic variation can destabilize generated filters and cause inaccurate segmentation.
  • Video Semantic Segmentation: Video semantic segmentation extends image segmentation to videos by modeling temporal continuity through cross-frame relations, while semi- or weakly supervised methods reduce annotation requirements.Methods use optical-flow-guided aggregation, sequential temporal propagation, image tags, labeled and unlabeled frames, or teacher-student pseudo-labeling.

4 VIDEO SEGMENTATION DATASETS

The survey reviews representative video segmentation datasets spanning object, semantic, instance, interactive, and language-guided settings, while comparing their annotation scope and benchmarking roles.

  • Twenty commonly used datasets are illustrated and their essential features are summarized for detailed review.The survey organizes dataset descriptions and statistics across its dataset section.
  • AVOS/SVOS/IVOS datasets: Youtube-Objects, FBMS59, and DAVIS16 support instance-agnostic AVOS and SVOS evaluation, whereas DAVIS17 adds instance-level annotations for AVOS, SVOS, and IVOS.DAVIS17 is comprehensive across these settings but relatively small in scale.
  • AVOS/SVOS/IVOS datasets: YouTube-VOS is the largest VOS dataset but supports only SVOS benchmarking.
  • Language-guided datasets: A2D Sentence contains 3,782 videos, 8 action classes, 6,655 sentences, and segmentation masks on 3 to 5 frames per video.Its train/test split is 3,017/737 videos.
  • VSS and VIS datasets: CamVid, CityScapes, NYUDv2, and VSPW are built for video semantic segmentation benchmarking, while BDD100K provides both VSS and VIS annotations.

5 PERFORMANCE COMPARISON

The survey benchmarks reviewed methods using task-specific datasets and metrics, with quantitative evaluation covering region overlap, boundaries, temporal stability, and runtime.

  • Benchmark scores are gathered from original articles, while most running speeds are measured by executing implementations on an RTX 2080Ti GPU.Values are borrowed from original papers for methods whose implementations are unavailable or poorly organized.
  • Evaluation metrics: Region Jaccard J measures intersection-over-union between predicted and ground-truth segmentation masks.It divides the intersection pixel count by the union size.
  • Evaluation metrics: Boundary accuracy F is the harmonic mean of boundary precision Pc and recall Rc, measuring contour agreement.F = 2PcRc/(Pc + Rc).
  • Evaluation metrics: Temporal stability T measures the pixel-level matching cost between successive segmentation boundaries while compensating for motion and small deformations.It does not penalize contour inaccuracies.
  • Object-level AVOS: RTNet reaches 85.6 region similarity J on DAVIS16 val, outperforming earlier deep learning-based methods such as SFL.

5.2 Instance-level AVOS Performance Benchmarking

Instance-level AVOS is evaluated on DAVIS17 using region and boundary metrics, while the survey reports UnOVOST as the top scorer among reviewed methods.

  • Metrics: Instance-level AVOS evaluation on DAVIS17 uses region Jaccard J, boundary accuracy F, and their mean J &F.Ground-truth and predicted object tracklets are matched using bipartite graph matching according to J &F.
  • Results: 67.9 J is achieved by UnOVOST, reported as the top instance-level AVOS score on DAVIS17 at the time of writing.

5.4 IVOS Performance Benchmarking

Interactive video object segmentation is benchmarked on DAVIS17 with time-aware accuracy criteria, while language-guided segmentation results on A2D Sentence show improvement from early to recent methods.

  • IVOS: Interactive video object segmentation on DAVIS17 is evaluated using AUC and J @60s.AUC summarizes the Time-versus-Jaccard curve, while J @60 measures accuracy under a 60-second budget.
  • IVOS: The method proposed by Cheng et al. is the top IVOS method in the summarized DAVIS17 results.
  • LVOS: Language-guided video segmentation uses overall IoU, mean IoU, Precision@K, and mAP over thresholds from 0.5 to 0.95.
  • LVOS: A2D Sentence is described as the most popular LVOS dataset, and recent solutions improve over the first LVOS model proposed in 2018.Runtime comparisons use 16-frame, 512×512 clips and 20-word textual sequences.

5.6 VSS Performance Benchmarking

On Cityscapes, IoU is the principal VSS metric, with category- and class-level reporting; EFC achieves the strongest reported class-level result.

  • Evaluation metric: IoU is the most widely used metric for video semantic segmentation, with Cityscapes reporting IoUcategory and IoUclass at two semantic granularities.Cityscapes evaluates 19 fine-grained classes grouped into 8 higher-level categories.
  • Results: 83.5% IoUclass is achieved by EFC on the Cityscapes validation set, the best result among eleven summarized VSS approaches.

5.7 VIS Performance Benchmarking

VIS evaluation on YouTube-VIS reports precision, mAP, and recall measures, and the surveyed results identify VisTR and Propose-Reduce as major state-of-the-art improvements.

  • Evaluation metric: VIS performance is evaluated using Precision@0.5, Precision@0.75, mAP over 0.50:0.05:0.95, and Recall@N.Metrics are evaluated per category and then averaged over the category set.
  • Results: Transformer-based VisTR and the redundant-sequence-proposal method Propose-Reduce greatly improve the state of the art on YouTube-VIS validation results.

5.8 VPS Performance Benchmarking

Video panoptic segmentation is evaluated with VPQ over temporal windows, and Cityscapes-VPS testing identifies ViP-DeepLab as the top method.

  • Evaluation metric: VPQ adapts image panoptic quality to video by matching predicted and ground-truth tubes with IoU above 0.5 and accumulating TP, FP, and FN by class and window.
  • Evaluation metric: VPQ is reported for temporal window sizes k∈{0, 5, 10, 15}, with k=1 equivalent to image panoptic quality.
  • Results: ViP-DeepLab is the top method on the Cityscapes-VPS test benchmark.

5.9 Summary

The survey identifies reproducibility and efficiency reporting gaps, while several established datasets have nearly reached performance saturation.

  • Reproducibility: Incomplete experimental setup, unavailable code or masks, and differing datasets and backbones make fair comparison impossible and hurt reproducibility.
  • Efficiency: Many AVOS, LVOS, and VPS methods omit execution-time reporting, and almost none report memory use despite application settings with limited computational resources.
  • Datasets: Performance on DAVIS16, DAVIS17, and A2D Sentence has nearly reached saturation, while newer datasets may add difficulty without identifying which challenges remain unsolved.

6 FUTURE RESEARCH DIRECTIONS

The survey identifies long-term and open-world video segmentation as important directions for practical deployment. It also calls for segmentation models with more flexible, input-dependent computation.

  • Long-Term Video Segmentation: Long-term video segmentation remains largely unexamined because existing datasets often span only several seconds.The proposed setting includes minute-level sequences and would increase demands on model re-detection.
  • Open World Video Segmentation: Open-world video semantic segmentation must adapt to unseen categories rather than assume complete knowledge of object classes.This boundary is especially relevant to robotics, self-driving cars, and surveillance deployments.
  • Efficient and Adaptive Computation: Flexible architectures could reduce computation by selectively activating network components according to the input.The survey contrasts this direction with fixed feed-forward structures and alternating heavy and lightweight modes.

7 CONCLUSION

The survey comprehensively reviews recent progress in video segmentation and organizes extensive literature, datasets, and benchmarks. It concludes that deep learning has produced notable progress while substantial challenges remain.

  • Survey Scope: The survey summarizes more than 150 deep learning models using task settings, technique contributions, and learning strategies.It also structures the field around 20 widely used datasets and benchmarking results on 7 widely used datasets.
  • Benchmarking: Benchmarking results from 7 widely used datasets support the survey’s structured review of video segmentation methods.The paper also presents a structured survey of 20 widely used video segmentation datasets.
  • Conclusion: Video segmentation has achieved notable progress with deep learning, but several challenges still lie ahead.
Loading 2107.01153v4…