Source-linked AI summary
The 2017 DAVIS Challenge on Video Object Segmentation
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, Luc Van Gool
TL;DR
Video object segmentation lacked a sufficiently challenging, standardized benchmark after DAVIS 2016 approached saturation. The paper introduces DAVIS 2017 as a larger multi-object dataset with evaluation metrics, a public competition, and workshop, reporting broad advances among participating methods. Its evaluation excludes temporal instability because frequent heavy occlusions make that measure less significant.
Problem
DAVIS 2016 was approaching saturation, motivating a larger and more challenging benchmark for video object segmentation.
Method
The paper constructs DAVIS 2017, defines region and boundary evaluation measures, and hosts a public competition with restricted test annotations and server-based evaluation.
Results
The challenge received entries from 22 teams and improved the state of the art by 20%, while methods ranged from 49.7% to 69.9% relative to the OSVOS baseline.
Takeaways & Limitations
DAVIS 2017 establishes a public benchmark for more difficult multi-object video segmentation and provides detailed comparisons of participating methods.
Takeaways & Limitations
Temporal instability is excluded from the main evaluation because frequent heavy occlusions make its results less significant, though reporting it on selected sequences is encouraged.
Abstract
from arXiv · showhide
We present the 2017 DAVIS Challenge on Video Object Segmentation, a public dataset, benchmark, and competition specifically designed for the task of video object segmentation. Following the footsteps of other successful initiatives, such as ILSVRC and PASCAL VOC, which established the avenue of research in the fields of scene classification and semantic segmentation, the DAVIS Challenge comprises a dataset, an evaluation methodology, and a public competition with a dedicated workshop co-located with CVPR 2017. The DAVIS Challenge follows up on the recent publication of DAVIS (Densely-Annotated VIdeo Segmentation), which has fostered the development of several novel state-of-the-art video object segmentation techniques. In this paper we describe the scope of the benchmark, highlight the main characteristics of the dataset, define the evaluation metrics of the competition, and present a detailed analysis of the results of the participants to the challenge.
1 INTRODUCTION
The 2017 DAVIS Challenge extends a successful video segmentation benchmark with a larger, more difficult dataset, public evaluation, competition, and workshop. It builds on DAVIS 2016, which had enabled strong algorithmic progress and was approaching saturation.
- Public benchmarks and competitions provide annotated data and fair evaluation environments that stimulate new computer vision algorithms and participation.
- DAVIS 2016 contains 50 high-definition sequences with pixel-accurate object masks on every frame and had reached around 80% performance for leading methods.
- The 2017 challenge introduces a larger, more challenging dataset with multiple annotated objects, plus a public competition and workshop.
- The new dataset contains 150 sequences, 10459 annotated frames, and 376 objects, with more distractors, occlusions, fine structures, small objects, and fast motion.
- The challenge received entries from 22 teams and brought a 20% improvement to the state of the art.
2 DATASET FACTS AND FIGURES
DAVIS 2017 expands video object segmentation from a single main object to semantically defined multiple objects and organizes evaluation through public and restricted dataset splits. The benchmark retains similar sequence lengths, uses 480p challenge images, and increases scene complexity.
- DAVIS 2017 segments multiple objects according to semantics, separating carried objects such as bags or skis even when they share motion with people or animals.
- The dataset provides public train and validation annotations, while test-dev and test-challenge expose only first-frame masks through an evaluation server.
- Train and validation sets extend DAVIS 2016 with applicable multiple-object annotations, while new sequences average around 3 objects and remain around 70 frames long.
- Figure 2 contrasts DAVIS 2016 single-object masks with DAVIS 2017 multiple-object masks, showing semantic distinctions despite identical object motion.
- Although many new sequences are captured at 4k resolution, the challenge evaluates downsampled 480p images to facilitate processing.
3 TASK DEFINITION AND EVALUATION METRICS
DAVIS 2017 evaluates semi-supervised video object segmentation from first-frame object masks, using region and boundary accuracy averaged over annotated object instances. The challenge ranking combines mean region similarity and contour accuracy, while temporal instability is excluded because heavy occlusions make it unreliable.
- Task definition: The task provides first-frame masks for identified objects and requires indexed masks for those objects throughout the remaining video frames.This is the semi-supervised setting, excluding interactive and unsupervised segmentation.
- Evaluation metrics: Region accuracy J is the intersection-over-union between predicted and ground-truth object masks.It divides the intersection area by the union area.
- Evaluation metrics: Boundary accuracy F measures boundary-pixel matching through the precision and recall of a bipartite match.The final boundary score is the F measure of matching precision and recall.
- Evaluation metrics: Temporal instability T is discarded because heavy occlusions strongly affect its behavior, although reporting it on selected sequences remains encouraged.The authors retain T as an informative stability measure outside the challenge ranking.
- Evaluation metrics: Overall performance averages J and F over all annotated object instances rather than averaging per-sequence scores.Per-sequence results are reported for information, but the overall metric is object-based.
- Evaluation metrics: The challenge ranking combines the object-level mean region and contour accuracies.The ranking metric is defined from m(J,S) and m(F,S).
4 ANALYSIS OF THE RESULTS
The challenge results reveal broad variation across methods, with remaining errors concentrated in foreground segmentation, object identity, and small-object handling. Complementary analyses show that combining methods and excluding tiny objects can substantially raise performance.
- The winner combines MaskTrack with a re-identification module that recovers objects when mask propagation fails.
- The nine methods range from 49.7% for the OSVOS baseline to 69.9%, while the winner leads the runner-up by 2.1%.
- Sequence difficulty varies widely: some objects are easy for all techniques, some challenge everyone, and others produce large performance differences.
- Combining the first two techniques yields a 5.4% boost over the first technique, while oracle combinations reach 75.3%, 76.7%, 77.4%, and 77.9% using up to two through five techniques.
- Foreground-background evaluation raises the winner's result from 69.9% to 82.4%, supporting difficulty in separating multiple foreground objects.
- False negatives dominate false positives and identity switches, while small objects are generally harder and can strongly affect multi-object performance.For the winner, removing objects below 5% of image area raises quality to 85% (+15%); the fourth method rises only to +8%.
- Misclassified-pixel percentages are not directly comparable to J and F because J and F also depend on object size.
5 CONCLUSIONS
The paper presents DAVIS 2017 as an expansion of DAVIS with more videos and challenging multi-object scenarios. It also establishes the challenge metrics and competition and analyzes participating teams' results.
- DAVIS 2017 extends DAVIS with more videos and more challenging scenarios, especially sequences containing multiple objects.
- The authors define evaluation metrics and competitions for a public challenge held with CVPR 2017, then analyze participating teams' results.