Source-linked AI summary
FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos
Suyog Dutt Jain, Bo Xiong, Kristen Grauman
TL;DR
Generic video object segmentation requires category-independent pixel masks without human intervention, but existing methods lack a unified treatment of appearance and motion and large-scale pixel-labeled video data. FusionSeg addresses this with a two-stream fully convolutional network and bootstrapped supervision, substantially improving state-of-the-art performance across challenging datasets.
Problem
Fully automatic segmentation must delineate generic foreground objects across video frames despite occlusions, shape changes, camera motion, and scarce large-scale pixel-labeled video data.
Method
FusionSeg uses separate fully convolutional appearance and optical-flow motion streams, fuses their predictions, and trains the motion stream from weakly annotated videos plus image segmentation annotations.
Results
The method substantially advances state of the art on multiple challenging datasets and sometimes outperforms methods requiring manual intervention.
Takeaways & Limitations
Learning appearance and motion jointly yields substantially stronger results than using either signal alone in the reported video segmentation experiments.
Takeaways & Limitations
Human-annotated data is required during training, although testing on new videos is fully automatic.
Abstract
from arXiv · showhide
We propose an end-to-end learning framework for segmenting generic objects in videos. Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos. We formulate this task as a structured prediction problem and design a two-stream fully convolutional neural network which fuses together motion and appearance in a unified framework. Since large-scale video datasets with pixel level segmentations are problematic, we show how to bootstrap weakly annotated videos together with existing image recognition datasets for training. Through experiments on three challenging video segmentation benchmarks, our method substantially improves the state-of-the-art for segmenting generic (unseen) objects. Code and pre-trained models are available on the project website.
1. Introduction
Fully automatic video object segmentation seeks category-independent pixel masks across frames, despite occlusions, shape changes, camera motion, and limited pixel-labeled video data. FusionSeg combines appearance and motion in an end-to-end framework and trains with weakly annotated videos plus image segmentation data.
- Video object segmentation separates foreground objects from background across frames using dense pixel-level masks that form space-time tubes.The task supports applications including activity recognition, object recognition, video editing, and rotoscoping.
- Fully automatic methods must handle generic objects without human involvement while producing accurate space-time boundaries under occlusions, shape changes, and camera motion.
- Appearance and motion provide complementary information, but existing methods had not unified them effectively for video segmentation.Motion can involve multiple object motions, camera or background motion, and small informative displacements.
- FusionSeg uses a two-stream fully convolutional network to fuse generic appearance and optical-flow motion cues into binary object-versus-background masks for each frame.The model segments both static and moving objects in new videos without human involvement.
- Weakly annotated videos and strongly labeled images provide training data when large-scale pixel-level video segmentation datasets are unavailable.The framework uses weak video labels together with existing image segmentation annotations.
- The method reports stronger results than appearance-only or motion-only alternatives, advances state of the art on multiple challenging datasets, and sometimes surpasses methods requiring manual intervention.
2. Related Work
Prior work includes fully automatic, human-guided, and category-specific segmentation approaches, but FusionSeg targets generic objects with dense video tubes and combines spatial and temporal cues through learned motion and appearance representations.
- Automatic methods: Fully automatic methods include supervoxels and proposals that generate space-time regions or object-like segments, but these do not straightforwardly provide accurately selected object-boundary tubes.
- Human-guided methods: Human-guided methods use annotations or corrections on selected frames, with motion mainly propagating labels or enforcing temporal consistency.
- Category-specific semantic segmentation: Category-specific semantic segmentation methods use end-to-end fully convolutional image architectures, whereas FusionSeg aims at generic objects whether or not their categories appear during training.
- Category-specific semantic segmentation: The paper identifies the lack of large-scale human-segmented video benchmarks as a bottleneck and uses weakly labeled video to fuse spatial and temporal cues.
- Deep learning with motion: Deep motion-appearance learning has helped classification, action recognition, tracking, and optical flow, but this work presents a fully automatic deep framework for video object segmentation.
3. Approach
FusionSeg builds separate generic appearance and motion streams, bootstraps motion supervision from weak video annotations, and fuses both streams to segment foreground objects automatically.
- Problem formulation: The task is posed as dense binary labeling: each pixel in every video frame is assigned to object or background.
- Pipeline: The pipeline trains an appearance model, uses it to bootstrap motion supervision, and finally fuses the two streams for video segmentation.
- Appearance Stream: The appearance stream uses an objectness-oriented fully convolutional network initialized from classification and fine-tuned with relatively few pixel-labeled masks.The network produces an objectness map with the same size as an input video frame.
- Two-stream architecture: Parallel appearance and motion streams process RGB frames and color-coded optical flow, respectively, before joining in a fusion layer.
- Motion Stream: Weakly annotated videos supply bounding boxes that are refined and filtered using appearance predictions, bounding-box tests, and optical-flow tests to create motion-stream masks.The bounding-box overlap threshold is 75%, and regions with more than 95% foreground pixels are discarded.
- Design choices: Using optical flow instead of consecutive RGB frames is presented as a way to expose motion cues without requiring the joint model to discover them from raw frames.
- Fusion: The fusion model combines independent stream predictions through separate branches and an element-wise multiplication branch that captures joint confidence.Pre-training the streams independently supports fusion with limited annotated video data and 24 additional trainable parameters.
4. Results
Across three challenging video object segmentation benchmarks, the method combines appearance and motion cues in a unified framework and substantially outperforms automatic and, in some cases, semi-supervised baselines.
- Evaluation setup: The evaluation covers DAVIS, YouTube-Objects, and Segtrack-v2 using Jaccard score, with comparisons against automatic and semi-supervised methods.The datasets contain diverse, challenging videos and the metric measures intersection-over-union overlap between predicted and ground-truth segmentations.
- Evaluation setup: At test time, the method requires no human involvement, although human-annotated data are used during training.This gives it the same test-time effort as automatic methods and less effort than semi-supervised methods.
- Ablations and baselines: The joint model combines appearance and motion streams, while the appearance-only and motion-only variants isolate each cue’s contribution.Flow-thresholding and flow-saliency provide additional motion-based baselines.
- Ablations and baselines: 77.8 Jaccard overlap is achieved on 100 labeled frames, compared with 70.2 before pruning with bounding boxes.These results indicate that the pseudo-ground-truth gathering strategy is effective.
- Ablations and baselines: Up to 11% absolute gain over individual streams comes from combining appearance and motion in the joint model.Both single-stream variants already perform well, with the motion-only variant relying purely on optical flow.
- Benchmark results: 71.51% overlap is achieved without human involvement, exceeding the semi-supervised BVS baseline’s 66.5% on DAVIS.The method also outperforms fully automatic methods and is reported to outperform existing methods across multiple challenging datasets.
- Benchmark results: On YouTube-Objects, the method exceeds NLC by 25% while NLC runs successfully on only 9% of the dataset, and it remains competitive with human-in-the-loop methods.Appearance performs especially well because the dataset shares categories with the PASCAL segmentation benchmark, while motion alone is weaker on static objects.
- Benchmark results: On Segtrack-v2, the method outperforms all semi-supervised and automatic methods except NLC, while the joint model still significantly boosts both single-stream variants.The weaker result relative to NLC may reflect Segtrack-v2’s low-quality, low-resolution videos.
5. Conclusions
The paper concludes that unified learning of motion and appearance yields stronger generic-object video segmentation while addressing practical training challenges. It identifies multiple touching foreground objects and intelligent human intervention as future extensions.
- Conclusions: The proposed approach achieves deeper synergy between motion and appearance for segmenting generic objects in video.The conclusion presents this synergy as one of the paper’s two central achievements.
- Conclusions: The approach also addresses practical challenges involved in training a deep network for video segmentation.The conclusion pairs this training contribution with the motion–appearance integration.
- Conclusions: The results show sizeable improvements over many existing methods, including some that require human intervention.Code and pre-trained models are made available on the project website.
- Future work: Future work will explore individuating multiple touching foreground objects and incorporating human intervention intelligently.These are proposed extensions rather than capabilities established by the current method.
6. Appendix
Per-video appendix results report how often the method surpasses the strongest comparison methods on DAVIS and Segtrack-v2.
- DAVIS: On DAVIS, the method outperforms the best per-video fully automatic and semi-supervised baseline in 25 of 50 videos.The appendix compares per-video results across all 50 DAVIS videos.
- Segtrack-v2: On Segtrack-v2, the method beats the best fully automatic method in 5 of 14 videos and semi-supervised HVS in 8 of 14 videos.The appendix reports these comparisons across the dataset’s 14 videos.