Source-linked AI summary
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation
Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokitsi Maninis, Luc Van Gool
TL;DR
The challenge addresses limited evaluation of unsupervised multi-object video segmentation and the need to reduce test-time human supervision. It introduces a track with revised annotations, new test sequences, formal definitions, rules, and metrics; the supplied passages report relatively low performance for an example method and define scope boundaries for annotation.
Problem
Existing work gave little attention to multi-object video segmentation without human input, while inconsistent annotations limited unsupervised evaluation.
Method
The challenge introduces an unsupervised multi-object track, re-annotates DAVIS 2017 train and validation sets, adds 60 test sequences, and specifies task definitions, rules, and evaluation metrics.
Results
An RVOS zero-shot configuration generating 20 video object proposals per sequence achieved relatively low performance on validation and test-dev sets compared with semi-supervised segmentation.
Takeaways & Limitations
The challenge extends DAVIS to evaluate video object proposals without test-time human input while retaining semi-supervised and interactive tracks.
Takeaways & Limitations
The annotation definition excludes people in crowds or backgrounds and assigns enclosed objects to enclosing objects, while carried items remain separate objects.
Abstract
from arXiv · showhide
We present the 2019 DAVIS Challenge on Video Object Segmentation, the third edition of the DAVIS Challenge series, a public competition designed for the task of Video Object Segmentation (VOS). In addition to the original semi-supervised track and the interactive track introduced in the previous edition, a new unsupervised multi-object track will be featured this year. In the newly introduced track, participants are asked to provide non-overlapping object proposals on each image, along with an identifier linking them between frames (i.e. video object proposals), without any test-time human supervision (no scribbles or masks provided on the test video). In order to do so, we have re-annotated the train and val sets of DAVIS 2017 in a concise way that facilitates the unsupervised track, and created new test-dev and test-challenge sets for the competition. Definitions, rules, and evaluation metrics for the unsupervised track are described in detail in this paper.
1 INTRODUCTION
The 2019 DAVIS Challenge builds on expanding video segmentation benchmarks and introduces an unsupervised multi-object track. This track addresses limited prior attention to multiple-object segmentation without human input and annotation burdens, supported by revised annotations.
- The DAVIS initiative expanded benchmark size and quality, prompting deep-learning methods that improved video object segmentation.
- The 2017 challenge further increased sequences, annotated frames, objects, and scenario difficulty, advancing accuracy and computational efficiency.
- The 2019 challenge introduces a new unsupervised track alongside the existing semi-supervised and interactive tracks.
- Prior work addressed video object segmentation without human input, but relatively little attention covered segmenting multiple objects.
- Unsupervised methods remove human effort, while object annotation is described as a time-and-effort bottleneck for real-time methods.
- DAVIS 2017 train and validation sets were re-annotated to resolve semantic inconsistencies that are problematic when no target objects are specified.
2 SEMI-SUPERVISED VIDEO OBJECT SEGMENTA-
The semi-supervised track retains its prior setup: methods receive first-frame masks for objects of interest and segment those objects in subsequent frames using unchanged dataset splits.
- The semi-supervised track remains unchanged from earlier editions, providing each object’s first-frame mask for subsequent-frame segmentation.
- The track uses train, validation, test-dev, and test-challenge splits of 60, 30, 30, and 30 sequences, respectively.
- The test-dev evaluation server accepts unlimited submissions, while the passage distinguishes it from test-challenge evaluation availability.
- Detailed evaluation metrics are provided in the 2017 edition manuscript and on the challenge website.
3 INTERACTIVE VIDEO OBJECT SEGMENTATION
The interactive track retains its 2018 procedure while adopting J &F evaluation and allowing an optional participant-selected frame choice for additional scribbles.
- The interactive track starts with human scribbles, predicts masks for all video frames, and exchanges results with a Web Service for further interaction.
- The evaluation metric changes from J alone to J &F, while the rest of the track remains based on the 2018 edition.
- The Web Service returns extra scribbles in poorly predicted regions, after which methods refine all-frame segmentation predictions.
- Participants may optionally select frames for subsequent scribbles instead of relying on the frame with the worst prediction.
- The interaction process is supported by the Python package released for the 2018 edition.
4 UNSUPERVISED VIDEO OBJECT SEGMENTATION
The paper defines unsupervised multi-object video segmentation around semantically selected, consistently visible objects and introduces re-annotated DAVIS data plus proposal-matching evaluation for the new track.
- Task definition: Object semantics take priority over shared motion, so a person and a carried bag remain separate objects despite similar motion.This differs from definitions based primarily on common motion or motion-connected instances.
- Task definition: The task redefines unsupervised segmentation for multiple objects by specifying which scene objects to segment and how to group them.The definition emphasizes objects likely to capture human attention and uses semantic object identity alongside temporal consistency.
- Evaluation: Predictions for objects absent from the ground-truth annotations are not penalized because some visually important objects remain difficult to distinguish consistently.This evaluation choice addresses ambiguity in deciding which objects would capture human attention.
- Evaluation: Methods output N non-overlapping video object proposals with identities consistent across frames, and each ground-truth object is matched to one proposal.The matching uses a one-to-one assignment and maximizes J &F through maximum-weight bipartite matching solved with the Hungarian algorithm.
- Evaluation: Final performance averages the optimally matched object scores, avoiding different weighting for sequences with different numbers of objects.The baseline RVOS experiment generates 20 proposals per sequence, and its relatively low performance versus semi-supervised segmentation highlights the task’s difficulty.
- Dataset: The DAVIS 2017 Unsupervised benchmark re-labels train and validation masks and adds new test-dev and test-challenge sets for the unsupervised track.The original train and validation sequences are retained, while their annotations are changed to satisfy the unsupervised definition.
5 CONCLUSIONS
The 2019 DAVIS Challenge introduces a new unsupervised multi-object video segmentation track alongside semi-supervised and interactive tracks. It also standardizes annotations, adds new evaluation sequences, and specifies task rules and metrics.
- The challenge includes semi-supervised, interactive, and unsupervised video object segmentation tracks.
- The new unsupervised multi-object track is accompanied by a task definition and consistent re-annotations of the DAVIS 2017 train and validation sets.
- The competition adds 60 sequences across new test-dev and test-challenge sets for the unsupervised track.
- The paper provides the rules and evaluation metrics for the unsupervised competition task.