Source-linked AI summary
Self-supervised Video Object Segmentation by Motion Grouping
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, Weidi Xie
TL;DR
The paper asks whether motion cues can support scalable video object segmentation without manual annotations, especially when appearance is unreliable. It learns foreground and background layers from optical flow with a slot-attention-based generative model. The approach outperforms prior self-supervised methods on public benchmarks and compares favorably with heavily supervised methods on camouflaged animals.
Problem
Appearance-based video segmentation requires extensive annotations and can fail when objects are visually indistinguishable from their backgrounds.
Method
A slot-attention-based generative model decomposes optical flow into foreground and background layers and trains through flow reconstruction and temporal consistency without manual annotations.
Results
The method outperforms previous self-supervised approaches on public benchmarks and compares favorably with heavily supervised video-segmentation approaches on MoCA.
Takeaways & Limitations
Motion cues can support self-supervised segmentation in realistic videos, including challenging camouflage scenes where appearance is unreliable.
Takeaways & Limitations
The two-slot model merges multiple independently moving objects into one layer, and flow-only input is limited when optical flow is uninformative, incomplete, noisy, or low-quality.
Abstract
from arXiv · showhide
Animals have evolved highly functional visual systems to understand motion, assisting perception even under complex environments. In this paper, we work towards developing a computer vision system able to segment objects by exploiting motion cues, i.e. motion segmentation. We make the following contributions: First, we introduce a simple variant of the Transformer to segment optical flow frames into primary objects and the background. Second, we train the architecture in a self-supervised manner, i.e. without using any manual annotations. Third, we analyze several critical components of our method and conduct thorough ablation studies to validate their necessity. Fourth, we evaluate the proposed architecture on public benchmarks (DAVIS2016, SegTrackv2, and FBMS59). Despite using only optical flow as input, our approach achieves superior or comparable results to previous state-of-the-art self-supervised methods, while being an order of magnitude faster. We additionally evaluate on a challenging camouflage dataset (MoCA), significantly outperforming the other self-supervised approaches, and comparing favourably to the top supervised approach, highlighting the importance of motion cues, and the potential bias towards visual appearance in existing video segmentation models.
1. Introduction
The paper addresses motion-based object segmentation without manual annotations, motivated by the limits of appearance-only segmentation in cluttered and camouflaged scenes. It groups pixels with similar optical flow into foreground and background layers using a self-supervised generative model.
- Appearance-only segmentation can fail when objects are visually indistinguishable from their backgrounds, especially in camouflage.
- The method exploits motion cues obtainable from unlabeled videos, requiring zero human annotation for training.
- A Transformer variant with slot attention groups pixels moving in similar directions at similar rates into shared layers.
- The model decomposes optical flow into homogeneous foreground and background layers and reconstructs the input flow.
- Across public benchmarks, the approach outperforms prior self-supervised models and approaches supervised performance, while improving substantially on camouflage data.
2. Related Work
Related work spans video object segmentation, motion segmentation, optical flow, Transformers, layered representations, and object-centric learning. The paper positions its contribution as applying slot-attention-style object-centric representations to realistic videos through motion rather than appearance.
- Unsupervised video object segmentation discovers targets without first-frame specification, while motion segmentation specifically discovers moving objects.
- Prior motion-segmentation methods cluster similar motion patterns, map motion to masks, or exploit independence between object and background motion.
- Optical flow is used across vision tasks and can sometimes substitute for RGB appearance cues, including in tracking.
- The method draws on slot attention, a Transformer variant that learns object-centric representations through iterative attention and query competition.
- Prior object-centric approaches were demonstrated on synthetic datasets, whereas this paper applies them to realistic videos using motion and discarding complex appearance textures.
3. Method
The method learns to segment optical flow by encoding it, iteratively binding pixels to foreground or background queries, and decoding layered reconstructions. Self-supervision combines flow reconstruction, mask entropy regularization, instance normalization, and temporal consistency across frame gaps.
- Flow Segmentation Architecture: The model predicts foreground and background layers with opacity masks, then linearly composites them to reconstruct the input optical flow.
- Flow Segmentation Architecture: A CNN encoder, iterative binding module, and CNN decoder respectively extract features, assign pixels to motion groups, and produce full-resolution layers.
- Iterative Binding: Iterative binding uses learnable queries and slot-attention-style competition to group pixels with similar motion into separate entities.
- Training Objectives: The reconstruction loss compares input and reconstructed flow, while entropy regularization encourages near-binary masks.
- Architectural Design: Instance normalization helps separate dominant background motion from foreground motion without explicitly estimating camera motion.
- Temporal Consistency: Temporal consistency requires matching predictions across flow fields computed with multiple frame gaps, although inference uses a single-step flow.
- Discussion: Learnable embeddings, instance normalization, and temporal consistency are identified as critical architectural changes for successful training.
4. Experimental Setup
Experiments evaluate pixelwise motion segmentation on DAVIS2016, SegTrackv2, and FBMS59, plus bounding-box localization on camouflaged animals in MoCA. The setup compares optical-flow extractors and uses standard dataset-specific annotation protocols.
- Datasets: DAVIS2016 contains 50 sequences and 3455 high-resolution frames with pixelwise annotations for predominantly moving objects.
- Datasets: SegTrackv2 contains 14 sequences with 976 annotated frames and challenges including blur, deformation, occlusion, slow motion, and interacting objects.
- Datasets: FBMS59 contains 59 sequences and 720 annotated frames, including multiple objects that may remain static temporarily.
- Datasets: MoCA provides 141 HD sequences of camouflaged animals, filtered to 88 sequences and 4803 frames with locomotion.
- Metrics: Pixelwise benchmarks use mean region similarity Jaccard, while MoCA uses detection success rate across IoU thresholds.
- Implementation: The study evaluates PWC-Net, RAFT, and self-supervised ARFlow, using different frame-gap ranges for most datasets and FBMS59.
5. Results
The method improves unsupervised moving-object segmentation, with ablations identifying important design choices and evaluations spanning standard, multi-object, and camouflage settings. It also emphasizes high runtime efficiency while documenting limitations of flow-only, two-slot segmentation.
- 5.1. Ablation Studies: With PWC-Net flow, Ours-A outperforms CIS by about 4.5 points on mean Jaccard, while RAFT provides further gains.
- 5.1. Ablation Studies: Instance normalization consistently improves performance, and five grouping iterations outperform three; eight iterations fail to converge within the same training budget.
- 5.1. Ablation Studies: Removing temporal consistency causes significant degradation, while entropy regularization also contributes importantly to performance.
- 5.2. Comparison with State-of-the-art: The method improves upon CIS by 9.1% on DAVIS2016 while nearing supervised models trained on thousands of images without pixel-level annotations.
- 5.2. Comparison with State-of-the-art: The model runs at over 80fps using a 4.77M-parameter VGG-style network and omitting computationally costly post-processing.
- 5.2. Comparison with State-of-the-art: On SegTrackv2 and FBMS59, flow-only input remains challenging when videos contain multiple objects but only some are moving; the method nevertheless achieves competitive performance.
- 5.3. Camouflage Breaking: On MoCA, the method significantly outperforms CIS, supervised COD and COSNet comparisons, while qualitative results show failures from noisy flow and partially moving animals.
- 5.4. Limitations: The current two-slot model assigns multiple independently moving objects to one layer, limiting separation beyond foreground and background.
6. Conclusion
The paper presents a self-supervised motion-segmentation model that uses only optical flow and no manual annotations. It surpasses previous self-supervised methods on public benchmarks and compares favorably with supervised approaches on MoCA.
- The model takes only optical flow as input and is trained without manual annotation for motion segmentation.
- It surpasses previous self-supervised methods on public benchmarks such as DAVIS2016 while narrowing the gap with supervised methods.
- On MoCA, the model compares favorably with top video-object-segmentation approaches trained with heavy supervision.
A. Training Details
The paper provides implementation details intended to make its results reproducible, including architectures, pseudocode, and hyperparameters.
- The reproduction details include the model architectures used in the experiments.
- The paper includes pseudocode to document the procedure for reproducing its results.
- The paper reports hyperparameters as part of its reproducibility documentation.
A.1. Encoder & Decoder
The architecture is documented through a network-architecture table and a separate pseudocode description of the iterative binding module. Training uses a specified batch size, learning-rate schedule, loss-weight schedule, and iteration count.
- The network architecture is specified in Table 4, while the iterative binding module is documented separately in Figure 5 pseudocode.
- All convolutions use padding 2 to preserve spatial resolution and are followed by instance normalization and ReLU, except the final layer.
- Training uses batch size 64, corresponding to 32 optical-flow pairs for consistency-loss training.
- The initial learning rate is 5e-4 with 200 warmup steps, then halves every 8e4 iterations.
- The entropy and consistency-loss scales increase fivefold during learning-rate decay, and training lasts about 300k iterations.
B. MoCA dataset curation
The MoCA videos are curated and transformed to produce a segmentation-oriented dataset with dense annotations and motion-focused filtering. The resulting release contains 88 sequences and 4803 frames.
- MoCA contains 141 high-definition videos averaging 11 seconds, depicting 67 kinds of camouflaged animals in natural scenes.
- The dataset is modified by removing channel logos and borders, resizing images to 720 x 1280, and adjusting annotations.
- Videos are sampled every three frames, up to 100 frames per video, to reduce duplicated consecutive frames from 24-fps resampling.
- Linear interpolation generates bounding boxes for missing frames, producing dense frame-wise annotations.
- Videos with predominantly absent locomotion or substantial non-primary-object motion are filtered out.
- The curated dataset contains 88 video sequences and 4803 frames and is intended for fair comparison.
C. Results breakdown
The results breakdown reports Jaccard-based segmentation evaluation across benchmark datasets, with predictions produced at 128 × 224 pixels and upsampled for comparison.
- Jaccard score measures intersection-over-union between predicted and ground-truth masks.
- Results are broken down by category on DAVIS2016, SegTrackv2, FBMS59, and MoCA.
- 128 × 224-pixel predictions are directly upsampled to original resolution before comparison with ground truth.
D. Qualitative results
The paper presents qualitative segmentation results across DAVIS2016, MoCA, SegTrackv2, and FBMS59, alongside sequence-wise and full-result tables.
- Table 5 reports sequence-wise results on SegTrackv2.
- Table 6 reports full results on DAVIS2016, including Jaccard and contour-accuracy measures.
- Table 7 reports sequence-wise results on FBMS59.
- Table 8 reports the results breakdown for MoCA.
- Figures 6 and 7 show qualitative results on DAVIS2016, MoCA, SegTrackv2, and FBMS59.