Source-linked AI summary
Collaborative Video Object Segmentation by Foreground-Background Integration
Zongxin Yang, Yunchao Wei, Yi Yang
TL;DR
Semi-supervised VOS has largely emphasized foreground embedding, leaving background information underused despite confusion among similar objects. CFBI learns collaborative foreground-background embeddings and matches references at both pixel and instance levels. It achieves state-of-the-art results on DAVIS 2016, DAVIS 2017, and YouTube-VOS without simulated data, fine-tuning, or post-processing.
Problem
Existing VOS methods pay little attention to background-region embedding and mainly develop foreground matching, creating a background-confusion problem for similar objects.
Method
CFBI learns contrastive foreground-background embeddings and combines pixel-level and instance-level matching through collaborative integration.
Results
CFBI outperforms other state-of-the-art methods on DAVIS 2016, DAVIS 2017, and YouTube-VOS, achieving 89.4%, 81.9%, and 81.4% J &F, respectively.
Takeaways & Limitations
Foreground and background features should be embedded collaboratively, while combining pixel-level and instance-level information improves robustness across object scales.
Takeaways & Limitations
VOS datasets contain far more background than foreground pixels, which can bias models toward background attributes during training.
Abstract
from arXiv · showhide
This paper investigates the principles of embedding learning to tackle the challenging semi-supervised video object segmentation. Different from previous practices that only explore the embedding learning using pixels from foreground object (s), we consider background should be equally treated and thus propose Collaborative video object segmentation by Foreground-Background Integration (CFBI) approach. Our CFBI implicitly imposes the feature embedding from the target foreground object and its corresponding background to be contrastive, promoting the segmentation results accordingly. With the feature embedding from both foreground and background, our CFBI performs the matching process between the reference and the predicted sequence from both pixel and instance levels, making the CFBI be robust to various object scales. We conduct extensive experiments on three popular benchmarks, i.e., DAVIS 2016, DAVIS 2017, and YouTube-VOS. Our CFBI achieves the performance (J$F) of 89.4%, 81.9%, and 81.4%, respectively, outperforming all the other state-of-the-art methods. Code: https://github.com/z-x-yang/CFBI.
1 Introduction
Semi-supervised VOS segments a target object through a video from its first-frame mask, but foreground-only embedding and matching can cause confusion among similar objects. CFBI integrates foreground and background embeddings with pixel- and instance-level matching, achieving state-of-the-art benchmark results without fine-tuning, simulated data, or post-processing.
- Motivation: Semi-supervised VOS segments a particular object across an entire video sequence using its first-frame mask.The task supports related applications including video instance segmentation and interactive video object segmentation.
- Motivation: Foreground-only embedding and matching can produce background confusion when videos contain similar objects, such as sheep, cars, people, or animals.The paper argues that background embeddings should be treated equally with foreground embeddings to relieve this confusion.
- CFBI: CFBI collaboratively learns foreground and background embeddings and combines pixel-level and instance-level information to handle different object scales.Its collaborative ensembler aggregates these information sources, while balanced random crops reduce bias toward background attributes during training.
- Results: 89.4% J &F on DAVIS 2016, 81.9% on DAVIS 2017, and 81.4% on YouTube-VOS outperform other state-of-the-art methods.These results use no simulated data, fine-tuning, or post-processing; testing augmentation further raises the scores to 90.1%, 83.3%, and 82.7%, respectively.
2 Related Work
Prior semi-supervised VOS methods either fine-tune at test time or avoid fine-tuning through learned matching and instance representations. CFBI extends this line by combining pixel- and instance-level embeddings with collaborative foreground-background integration.
- Semi-supervised Video Object Segmentation: Early semi-supervised VOS methods rely on first-frame fine-tuning, while later approaches seek better runtime by avoiding test-time fine-tuning.The related methods include online adaptation, optical-flow propagation, and combinations of multiple networks.
- Semi-supervised Video Object Segmentation: PML and VideoMatch learn pixel-wise embedding spaces for matching current-frame pixels to reference-frame pixels, while FEELVOS adds global and local matching.These approaches focus on learned pixel-level correspondence across frames.
- CFBI: CFBI combines pixel-level and instance-level embeddings and additionally learns background embeddings through collaborative integration.The framework uses attention mechanisms alongside its matching representations.
3 Method
CFBI integrates foreground and background embeddings with pixel-level and instance-level matching to address appearance confusion, object motion, and scale variation in semi-supervised video object segmentation.
- Collaborative Foreground-Background Integration: CFBI learns foreground and background embeddings collaboratively, encouraging contrastive representations that reduce confusion between similar foreground and background appearances.The method explicitly redesigns pixel distance using trainable foreground and background biases.
- Collaborative Pixel-level Matching: CFBI combines global and multi-local pixel matching, using multiple neighborhood sizes to handle variable object motion between adjacent frames.The network learns to select an appropriate local scale, while the largest-window intermediate results limit added computation.
- Collaborative Pixel-level Matching: The collaborative pixel-level output concatenates current-frame embeddings, previous-frame embeddings and mask, multi-local matching, and global matching maps.Introducing the previous embedding improves J &F by about 0.5%.
- Collaborative Instance-level Attention: CFBI adds instance-level attention for large objects by pooling foreground and background embeddings from the first and previous frames into one guidance vector.The resulting vector contains information from both temporal references and both foreground and background regions.
- Collaborative Ensembler: A collaborative ensembler aggregates foreground-background and pixel-level-instance-level information while using large receptive fields to relieve local ambiguities.Its design uses a downsample-upsample structure with Res-Blocks and an ASPP module.
4 Implementation Details
CFBI modifies training augmentation and sequence sampling to reduce foreground-background imbalance and support stable video object segmentation training.
- Balanced Random-Crop: Balanced random-crop addresses the foreground-background pixel imbalance that can bias models toward background attributes.It restricts the first-frame crop to contain enough foreground information.
- Balanced Random-Crop: Balanced random-crop applies the same cropped window to the first, previous, and current frames in a sampled sequence.The crop is repeatedly resampled until the first frame contains enough foreground pixels.
- Sequential Training: Training samples consecutive frame sequences, using a randomly selected reference frame and a continuous previous-current frame sequence.The first predicted frame uses the previous frame’s ground-truth mask, while later frames use the latest prediction.
- Training Details: The network uses DeepLabv3+ with a dilated ResNet-101 backbone and a depth-wise separable convolution for pixel-wise embeddings.The embedding convolution operates with stride 4.
- Training Details: Multi-local matching downsamples embedding features by half and uses window sizes K = {2, 4, 6, 8, 10, 12} to reduce GPU memory use.Group normalization and gated channel transformation are used in the collaborative ensembler for small-batch training stability and performance.
- Training Details: Training uses DAVIS 2017 and YouTube-VOS videos, SGD with momentum 0.9, and bootstrapped cross-entropy over the 15% hardest pixels.Batch-normalization parameters are frozen during training.
5 Experiments
Experiments evaluate CFBI on DAVIS and YouTube-VOS using standard segmentation metrics, qualitative cases, and component ablations. CFBI achieves strong benchmark results, while ablations support the contributions of foreground-background integration, multi-scale embeddings, and training strategies.
- Evaluation Settings: Experiments train and evaluate CFBI on YouTube-VOS and DAVIS, using J &F as the average of region-overlap and boundary-similarity scores.YouTube-VOS includes unseen object categories, making its validation set suitable for measuring generalization ability.
- YouTube-VOS: 82.2% on the YouTube-VOS Testing 2019 split exceeds the Rank 1 result of 81.8% without model ensembling.The comparison is reported for the unseen and average metrics.
- DAVIS 2016: 89.4% J &F on DAVIS 2016 validation exceeds STMVOS at 89.3% and FEELVOS at 81.7%, while maintaining comparable fast inference speed to FEELVOS.Multi-scale and flip augmentation increases the score to 90.1% but raises inference time to 9 seconds.
- DAVIS 2017: 81.9% on DAVIS 2017 validation exceeds FEELVOS at 71.5% and slightly surpasses STMVOS at 81.8% without simulated data.The score reaches 83.3% with evaluation augmentation, and CFBI exceeds STMVOS by 2.6% on the more challenging testing split.
- Qualitative Results: Qualitative results show successful tracking of similar-looking sheep and person-dog tracking after occlusion, but one hand is missed when people are similar, close, and blurred.The reported failure concerns one hand of the right person in the judo video.
- Ablation Study: Removing background embedding reduces the ablation result from 74.9% to 70.9%, while removing pixel-level or instance-level background information yields 73.0% or 72.3%.Removing foreground and background bias further reduces the result to 72.8%.
- Ablation Study: The full CFBI result is 74.9% versus 68.3% for the reproduced baseline; removing multi-local matching, sequential training, balanced cropping, or instance-level attention lowers performance.The reported alternatives produce 73.8%, 73.3%, 72.8%, and 72.7%, respectively, while replacing the collaborative ensembler performs 1.6% worse.
6 Conclusion
CFBI integrates foreground and background embeddings for video object segmentation, combining contrastive learning with pixel- and instance-level representations. The framework achieves state-of-the-art results while remaining simple and fast.
- CFBI introduces collaborative foreground-background integration for video object segmentation.It imposes contrastive feature embedding between the foreground target and corresponding background.
- Pixel-level and instance-level embeddings make CFBI robust to various object scales.The two embedding types support feature representations at different scales while keeping the network simple and fast.
- CFBI achieves new state-of-the-art results on three popular video object segmentation benchmarks.