Source-linked AI summary
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation
Weiyao Wang, Matt Feiszli, Heng Wang, Du Tran
TL;DR
Open-world applications need video segmentation that includes novel objects, but existing methods and datasets are built around predefined categories. UVO introduces a large, densely annotated benchmark for class-agnostic video segmentation, demonstrating its use for segmentation, tracking, and super-voxel evaluation while supporting broader video-understanding research.
Problem
Real-world video applications require segmenting novel as well as known objects, whereas current methods and datasets assume closed-world taxonomies.
Method
UVO constructs an open-world, class-agnostic video object-segmentation benchmark with exhaustive annotations and evaluates complementary top-down and bottom-up baselines.
Results
UVO provides 12.29 object annotations per video on average, 7x more than YouTube-VIS and YouTube-VOS and 4x more than DAVIS, and supports tracking and super-voxel evaluation.
Takeaways & Limitations
UVO serves as a testbed for open-world segmentation and broader video-understanding tasks beyond classification and detection.
Abstract
from arXiv · showhide
Current state-of-the-art object detection and segmentation methods work well under the closed-world assumption. This closed-world setting assumes that the list of object categories is available during training and deployment. However, many real-world applications require detecting or segmenting novel objects, i.e., object categories never seen during training. In this paper, we present, UVO (Unidentified Video Objects), a new benchmark for open-world class-agnostic object segmentation in videos. Besides shifting the problem focus to the open-world setup, UVO is significantly larger, providing approximately 8 times more videos compared with DAVIS, and 7 times more mask (instance) annotations per video compared with YouTube-VOS and YouTube-VIS. UVO is also more challenging as it includes many videos with crowded scenes and complex background motions. We demonstrated that UVO can be used for other applications, such as object tracking and super-voxel segmentation, besides open-world object segmentation. We believe that UVo is a versatile testbed for researchers to develop novel approaches for open-world class-agnostic object segmentation, and inspires new research directions towards a more comprehensive video understanding beyond classification and detection.
1. Introduction
Open-world object segmentation is needed because real applications encounter known and unknown objects, while existing closed-world methods and datasets cannot reliably represent or evaluate them. UVO addresses this gap with a larger benchmark and a more efficient annotation approach.
- Motivation: Real-world applications require exhaustive segmentation of both known and unknown objects encountered in videos.Examples include object search, instance registration, human-object interaction modeling, activity understanding, embodied AI, and augmented-reality assistants.
- Limitations of existing methods: Current detection and segmentation methods are designed for closed-world taxonomies and perform poorly on unseen objects.Top-down methods are biased toward seen-class contextual cues, while bottom-up methods based on local color or motion cues lack semantic object-boundary information.
- Dataset gap: Open-world evaluation requires exhaustive annotations because detecting unannotated objects is otherwise unrewarded or penalized.The paper notes that existing datasets generally lack exhaustive video annotations at scale.
- UVO benchmark: UVO introduces a benchmark for open-world object segmentation and provides comprehensive baselines for studying the task.The benchmark is intended as a testbed for research beyond classification and detection.
- UVO benchmark: UVO’s dataset-construction method uses object interpolation and tracking and is 4x more efficient than the baseline.The paper presents this method as part of its contribution toward scalable open-world dataset construction.
2. Related Work
Prior work spans closed-world image and video detection or segmentation datasets and open-world recognition methods. UVO builds on these lines while targeting exhaustive, taxonomy-free video object segmentation.
- Open-world recognition: Open-world recognition research studies how to identify new object categories after training on a closed-world dataset.Prior methods explicitly distinguish unknown from known objects, including through outlier detection in embedding space.
- Related datasets: Existing detection and segmentation benchmarks include major image datasets and newer video datasets such as DAVIS, YouTube-VOS, MOTS, YouTube-VIS, and TAO.These datasets provide the broader benchmark context from which UVO is developed.
3. Open-World Object Segmentation
UVO defines open-world video segmentation as exhaustive, class-agnostic segmentation of all entities, including unseen categories. It combines dense annotations, broad video diversity, and semi-automated propagation to create a challenging benchmark with substantial non-COCO coverage and motion variation.
- Open-world segmentation requires class-agnostic and exhaustive masks so models can detect unseen categories during testing.
- UVO provides dense, exhaustive annotations without a predefined taxonomy, averaging 12.29 objects per video—7x more than YouTube-VIS and YouTube-VOS and 4x more than DAVIS.Object counts follow a long-tail distribution, with nearly 100 instances in extreme videos.
- Dataset construction: The dataset uses diverse Kinetics videos and defines objects as non-background or non-stuff regions, with coarse meaningful segmentation and explicit rules for connected groups and human accessories.
- Efficient annotation: 11 minutes per frame is required by the proposed pipeline, compared with 45 minutes for naive annotation and linking and 30.7 minutes for copy-paste.
- Dataset characteristics: 57% of object instances fall outside the 80 COCO categories, while UVO also exhibits broader object lifespans and larger ranges and average magnitudes of motion than YouTube-VIS.UVO and YouTube-VIS have similar camera-motion distributions, but UVO contains more varied object motion.
4. Experiments
Experiments evaluate frame-level, video-level, bottom-up, and tracking baselines, showing that UVO is challenging, complementary to existing datasets, and sensitive to motion and annotation density.
- Baselines and implementation: UVO evaluations use Mask R-CNN for frame-level segmentation, MaskTrack R-CNN for video-level segmentation, GBH for bottom-up segmentation, and STM for tracking.MaskTrack R-CNN links objects across frames, GBH progressively groups similar super-voxels, and STM uses memory and attention for tracking.
- Cross-dataset evaluation: Models trained on COCO or YouTube-VIS suffer significant performance drops on UVO, while UVO finetuning improves results but remains below prior-dataset performance.LVIS models also perform poorly on UVO, and pretraining on LVIS is slightly worse than COCO.
- Compatibility and transfer: UVO finetuning improves YouTube-VIS performance by 1.3% on AR100, while UVO evaluation is slightly harder than existing tracking datasets.UVO does not cover 29 of COCO’s 80 categories, and tracking performance is lower except for J-score on unseen YouTube-VOS objects.
- Bottom-up segmentation: Super-voxel algorithms achieve 19.8% approximate AR103 on UVO, 2.6% higher than top-down MaskTrack R-CNN AR100, under a metric that is not directly comparable.The comparison is intended as an alternative open-world baseline rather than evidence that super-voxels are stronger or weaker.
- Motion analysis: Higher object motion and larger camera motion reduce video-model performance on both UVO and YouTube-VIS.Performance drops significantly for high object motion, and camera-motion effects are reported separately for both datasets.
- Annotation density: Training at 6fps and 30fps differs by 2.7% in AR100 when evaluated at 30fps, while sparse training with interpolated masks can approach dense-annotation performance using more videos.Models trained at both frame rates differ only slightly when evaluated at 6fps; two times more videos further closes the gap with 30fps annotations.
5. Conclusion
The paper presents UVO as a larger open-world object-segmentation benchmark for an unsolved problem with real-world applications. It positions UVO as a basis for broader video-understanding research.
- Conclusion: UVO is a new benchmark for open-world object segmentation, an unsolved problem with various real-world applications.The benchmark differs from current benchmarks in both its open-world setup and its larger size and annotation volume.
- Future directions: The authors believe UVO will enable research on long-term video modeling and complex video-understanding tasks.The planned release includes continued annotation, a test server, and a challenge workshop for open-world object segmentation.
A. Camera Motion Statistics and Ablations
The ablations examine camera motion and pretraining choices, finding that larger camera motion harms segmentation and ImageNet initialization outperforms Kinetics pretraining.
- Camera motion: Higher camera translation or rotation corresponds to worse MaskTrack R-CNN performance on both UVO and YouTube-VIS.The datasets contain broad camera-motion ranges; UVO has slightly higher rotations, while YouTube-VIS has higher average translations.
- Pretraining: Replacing ImageNet with Kinetics pretraining worsens all results for Mask R-CNN and MaskTrack R-CNN.The authors suggest Kinetics videos and taxonomy are less suitable for object-detector initialization because they are organized around human actions rather than objects.
B. Implementation Details
Finetuning on UVO uses fixed learning rates and iteration budgets for Mask R-CNN and MaskTrack R-CNN, with dataset interpolation implemented in PyTorchVideo.
- Finetuning: Mask R-CNN and MaskTrack R-CNN finetuning both use an initial learning rate of 5 × 10^-4, with 4.5k and 3240 iterations respectively.The dataset interpolation pipeline is implemented in PyTorchVideo.