Source-linked AI summary
RVOS: End-to-End Recurrent Network for Video Object Segmentation
Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques, Xavier Giro-i-Nieto
TL;DR
Multi-object video object segmentation is especially difficult in the zero-shot setting because no initial mask identifies the objects to track. RVOS addresses this with end-to-end spatial and temporal recurrence, supports both zero-shot and one-shot VOS, and reports strong benchmark comparisons without online learning.
Problem
Existing methods do not provide unified multi-object segmentation with full end-to-end temporal training and no post-processing.
Method
RVOS combines spatial recurrence for discovering object instances with temporal recurrence for maintaining their correspondence across frames.
Results
RVOS provides the first quantitative zero-shot VOS results on DAVIS-2017 and YouTube-VOS and outperforms prior methods without online learning.
Takeaways & Limitations
The same fully end-to-end recurrent architecture supports both zero-shot and one-shot multi-object VOS without post-processing.
Takeaways & Limitations
Zero-shot evaluation is limited because benchmarks may omit some objects appearing in videos; YouTube-VOS annotates up to five instances per video.
Abstract
from arXiv · showhide
Multiple object video object segmentation is a challenging task, specially for the zero-shot case, when no object mask is given at the initial frame and the model has to find the objects to be segmented along the sequence. In our work, we propose a Recurrent network for multiple object Video Object Segmentation (RVOS) that is fully end-to-end trainable. Our model incorporates recurrence on two different domains: (i) the spatial, which allows to discover the different object instances within a frame, and (ii) the temporal, which allows to keep the coherence of the segmented objects along time. We train RVOS for zero-shot video object segmentation and are the first ones to report quantitative results for DAVIS-2017 and YouTube-VOS benchmarks. Further, we adapt RVOS for one-shot video object segmentation by using the masks obtained in previous time steps as inputs to be processed by the recurrent module. Our model reaches comparable results to state-of-the-art techniques in YouTube-VOS benchmark and outperforms all previous video object segmentation methods not using online learning in the DAVIS-2017 benchmark. Moreover, our model achieves faster inference runtimes than previous methods, reaching 44ms/frame on a P100 GPU.
1. Introduction
RVOS addresses unified multi-object video object segmentation by combining spatial and temporal recurrence in a fully end-to-end architecture. It supports zero-shot and one-shot settings and reports benchmark results for DAVIS-2017 and YouTube-VOS.
- Motivation: Existing approaches often process frames independently or handle temporal information through optical flow or mask propagation.Some recurrent methods model spatio-temporal evolution but rely on optical flow or process only one object per pass.
- Method: RVOS uses recurrence in both spatial and temporal domains to discover instances within frames and maintain their correspondence across time.Spatial recurrence predicts multiple instance masks, while temporal recurrence preserves consistent instance ordering between frames.
- Motivation: RVOS targets video object segmentation, including the more challenging multi-object and zero-shot settings.Zero-shot segmentation provides no initial object masks, requiring the model to discover segments throughout the sequence.
- Method: The architecture performs multi-object video segmentation end to end without post-processing.The recurrent network learns to predict one mask per object instance and handles matching between instances across frames internally.
- Evaluation: RVOS adapts to both zero-shot and one-shot VOS and reports the first quantitative zero-shot results on DAVIS-2017 and YouTube-VOS.For one-shot VOS, masks from previous time steps are fed into the recurrent network.
- Evaluation: RVOS outperforms previous VOS methods that do not use online learning and is reported as the fastest method.The supplied introduction states this comparative result without specifying a numerical score.
2. Related Work
Prior VOS methods separately address spatial segmentation, temporal coherence, or end-to-end training, often relying on online learning, pretrained auxiliary tasks, or single-object processing. RVOS is presented as a unified end-to-end alternative for multiple objects without post-processing.
- Video object segmentation: VOS methods either model temporal evolution or predict segmentations independently for each frame.Temporal modeling seeks coherence across video frames, whereas frame-wise methods omit explicit temporal modeling.
- One- and zero-shot VOS: One-shot methods commonly use the first annotated frame to fine-tune a pretrained network or guide later predictions with object masks.Online learning adapts model weights using the initial frame and its masks.
- Temporal modeling: Temporal VOS approaches often depend on optical flow, pretrained models, or propagated masks, preventing fully end-to-end training in the cited methods.These approaches obtain temporal coherence through auxiliary motion information or prior-frame predictions.
- One- and zero-shot VOS: Zero-shot VOS does not require initialization, whereas one-shot VOS uses a single annotated frame to estimate the remaining sequence.The distinction reflects whether an initial object mask is available.
- End-to-end training: Existing end-to-end approaches generally trade off multiple-object processing against temporal modeling, with some requiring one forward pass per object and post-processing to merge instances.Frame-based methods support multiple objects, while cited temporally recurrent methods process a single object at a time.
- RVOS contribution: RVOS is presented as the first full end-to-end video-sequence model for multiple objects that requires no post-processing.The model is trained using video sequences and their masks rather than relying on models trained for other tasks.
3. Model
RVOS uses an encoder-decoder architecture with recurrent processing across object instances within each frame and across frames. Its inputs differ for one-shot and zero-shot VOS, while the decoder outputs temporally consistent object masks.
- Model inputs and outputs: One-shot VOS uses RGB frames plus object masks from the frame where each object first appears, whereas zero-shot VOS uses only RGB frames.
- Encoder: The encoder is a ResNet-101 pretrained on ImageNet and produces multi-resolution features for each video frame.
- Decoder: The decoder is a hierarchical recurrent stack of ConvLSTMs that predicts a mask sequence for object instances at each frame.
- Temporal recurrence: The decoder keeps object indices consistent across frames, outputs a constant number of predictions, and represents disappeared objects with empty masks.
- Decoder: Decoder states combine current-frame encoder features with preceding-layer, previous-object, and same-object previous-frame representations, plus the prior mask.
- Decoder initialization: The first spatial object state uses a zero matrix because no previous spatial hidden state exists for that object.
4. Experiments
Experiments evaluate RVOS across one-shot and zero-shot VOS on YouTube-VOS and DAVIS-2017, comparing recurrence configurations, state-of-the-art methods, qualitative behavior, and runtime. Spatio-temporal recurrence improves segmentation over spatial or temporal alternatives, while RVOS remains competitive without online learning and runs quickly.
- Experimental setup: The experiments evaluate one-shot and zero-shot VOS on YouTube-VOS and DAVIS-2017 using region similarity J and contour accuracy F.YouTube-VOS additionally separates scores for seen and unseen categories.
- One-shot video object segmentation: Spatio-temporal recurrence improves J and F for seen and unseen YouTube-VOS categories over spatial and temporal models in one-shot VOS.Training uses an 80%-20% split, and the inferred-mask variant RVOS-Mask-ST+ outperforms the ground-truth-mask-trained model.
- One-shot video object segmentation: RVOS-Mask-ST+ has comparable YouTube-VOS results to S2S without online learning and outperforms the other listed methods for seen categories.Comparisons for unseen categories are not fair against methods fine-tuned using validation-set first-frame annotations.
- One-shot video object segmentation: Fewer annotated instances produce better one-shot VOS results, with the best performance on sequences containing one or two objects.This analysis is reported for RVOS-Mask-ST+ using J and F.
- DAVIS-2017 benchmark: RVOS-Mask-ST+ pretrained on YouTube-VOS outperforms DAVIS-2017 methods without online learning, while fine-tuning further outperforms some online-learning methods.Direct application obtains J = 21.7 and F = 27.3; DAVIS-2017 fine-tuning obtains J = 23.0 and F = 29.9.
- Runtime analysis: RVOS achieves 44ms per frame on a P100 and 67ms per frame on a K80, remaining faster than OSMN at 44ms versus 65ms on a P100.The authors attribute the fast runtime partly to sharing the encoder forward pass across objects.
5. Conclusions
RVOS is a fully end-to-end trainable model for multiple-object VOS, designed for both one-shot and zero-shot settings and evaluated on YouTube-VOS and DAVIS-2017. Experiments report first zero-shot results on both benchmarks and improvements over spatial-only or temporal-only recurrence, plus one-shot gains over methods without online learning.
- RVOS is a fully end-to-end trainable model for multiple objects in video object segmentation, using recurrence across spatial and temporal domains.The model was designed for both one-shot and zero-shot VOS and tested on YouTube-VOS and DAVIS-2017.
- The experiments provide the first reported zero-shot VOS results on both YouTube-VOS and DAVIS-2017.
- Spatio-temporal recurrence improves models that consider only the spatial or only the temporal domain.
- For one-shot VOS, RVOS outperforms state-of-the-art techniques that do not use online learning on both benchmarks.
- The project code is available on the project website.