PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration

Zongxin Yang, Yunchao Wei, Yi Yang

arXiv:2010.06349v2cs.CV

TL;DR

Semi-supervised VOS must segment a target object across a video, but prior embedding methods largely focused on foreground features and could suffer background confusion. CFBI integrates foreground and background embeddings with pixel- and instance-level matching, and CFBI+ adds multi-scale and atrous matching. CFBI+ achieved 82.9% J&F on DAVIS 2017 and 82.8% on YouTube-VOS without simulated data, fine-tuning, or postprocessing.

  • Problem

    Prior semi-supervised VOS embedding methods focused mainly on foreground regions, leaving background embedding underexplored despite confusion from similar background objects.

  • Method

    CFBI collaboratively embeds foreground and corresponding background regions, combines pixel-level and instance-level matching, and extends this design in CFBI+ with multi-scale and Atrous Matching.

  • Results

    82.9% J&F on DAVIS 2017 and 82.8% on YouTube-VOS were achieved by CFBI+ without simulated data, fine-tuning, or postprocessing.

  • Takeaways & Limitations

    Collaborative foreground-background embedding, multi-scale matching, and atrous matching provide a robust and efficient VOS framework that the paper presents as a solid baseline.

Abstract

from arXiv · show

This paper investigates the principles of embedding learning to tackle the challenging semi-supervised video object segmentation. Unlike previous practices that focus on exploring the embedding learning of foreground object (s), we consider background should be equally treated. Thus, we propose a Collaborative video object segmentation by Foreground-Background Integration (CFBI) approach. CFBI separates the feature embedding into the foreground object region and its corresponding background region, implicitly promoting them to be more contrastive and improving the segmentation results accordingly. Moreover, CFBI performs both pixel-level matching processes and instance-level attention mechanisms between the reference and the predicted sequence, making CFBI robust to various object scales. Based on CFBI, we introduce a multi-scale matching structure and propose an Atrous Matching strategy, resulting in a more robust and efficient framework, CFBI+. We conduct extensive experiments on two popular benchmarks, i.e., DAVIS and YouTube-VOS. Without applying any simulated data for pre-training, our CFBI+ achieves the performance (J&F) of 82.9% and 82.8%, outperforming all the other state-of-the-art methods. Code: https://github.com/z-x-yang/CFBI.

1 INTRODUCTION

Semi-supervised VOS segments a specified object throughout a video from its first-frame mask. CFBI addresses background confusion by integrating foreground and background embeddings with pixel- and instance-level matching, while CFBI+ adds multi-scale and atrous matching for stronger, more efficient segmentation.

  • 1 INTRODUCTION: Semi-supervised VOS segments a particular object across an entire video using the object mask provided in the first frame.The task supports related applications including video instance segmentation and interactive video object segmentation.
  • 1 INTRODUCTION: Foreground-only embedding can confuse predictions when background scenes contain similar objects, motivating equal treatment of foreground and background regions.The paper illustrates this problem with similar sheep appearing in the background.
  • 1 INTRODUCTION: CFBI learns and matches embeddings for both foreground targets and corresponding background regions to relieve background confusion.The framework separates collaborative foreground-background processing from approaches that match only the foreground.
  • 1 INTRODUCTION: CFBI combines pixel-level and instance-level embeddings to cover different object scales, while pixel-level matching retains detailed object information.Pixel-level matching uses global and local mechanisms, whereas instance-level processing addresses larger-scale objects and pixel-wise noise.
  • 1 INTRODUCTION: 82.9% J&F on DAVIS 2017 and 82.8% on YouTube-VOS were achieved by CFBI+ without simulated data, fine-tuning, or postprocessing.The paper reports that CFBI+ outperformed other state-of-the-art methods and had faster multi-object inference than previous state-of-the-art methods.
  • 1 INTRODUCTION: CFBI+ extends CFBI with Atrous Matching and a multi-scale matching framework to improve matching efficiency and robustness.The extension also reports additional ablations, model-setting analyses, and visualization results.

2 RELATED WORK

Earlier semi-supervised VOS methods either fine-tuned at inference or avoided fine-tuning through learned matching and memory mechanisms. Related work explored instance-level and pixel-level foreground embeddings, while CFBI additionally incorporates background embedding and attention mechanisms.

  • 2 RELATED WORK: Early semi-supervised VOS methods fine-tuned on the first frame during evaluation, substantially slowing inference.Later approaches sought to avoid fine-tuning and improve runtime.
  • 2 RELATED WORK: OSMN explored instance-level matching, whereas PML, VideoMatch, and FEELVOS used pixel-level embeddings or matching to preserve target appearance diversity.The related methods represent different strategies for learning foreground features and matching current frames to reference information.
  • 2 RELATED WORK: Pixel-level matching can introduce noise when background pixels resemble the foreground, exposing a limitation of foreground-focused embedding strategies.This limitation motivates collaborative integration of background embedding in CFBI.
  • 2 RELATED WORK: CFBI extends prior embedding approaches by learning background features collaboratively and using both pixel-level and instance-level embeddings with attention mechanisms.The related-work discussion presents this as a response to limitations in prior foreground-focused methods.

3 METHODOLOGY

CFBI integrates foreground and background embeddings through collaborative pixel-level matching and instance-level attention, while CFBI+ extends this design with multi-scale matching and Atrous Matching for greater efficiency.

  • Collaborative video object segmentation by Foreground-Background Integration: CFBI extracts pixel-wise embeddings, separates reference embeddings into foreground and background pixels, and combines their matching with instance-level attention to guide prediction.The framework uses the first, previous, and current frames before generating a prediction through the Collaborative Ensembler.
  • Collaborative Pixel-level Matching: Foreground-background pixel-level matching adds background information to global and multi-local matching, including trainable foreground and background biases.The redesigned distance distinguishes pixels according to whether they belong to background or foreground sets.
  • Collaborative Pixel-level Matching: Multi-local matching uses different neighborhood scales because object motion between adjacent frames varies across sequences.Intermediate results from the largest window make the additional computational cost negligible.
  • Collaborative Instance-level Attention: Collaborative instance-level attention pools foreground and background embeddings from the first and previous frames into guidance that adjusts CE feature channels.The attention mechanism uses channel-wise average pooling and a learned gate to guide the Collaborative Ensembler.
  • CFBI+: Towards Efficient Multi-scale Matching: CFBI+ extracts features at strides S = 4, 8, 16, fuses them with an FPN, and applies CFBI matching at every scale before sending outputs to CE stages.The multi-scale structure is designed to support matching across different feature resolutions.
  • CFBI+: Towards Efficient Multi-scale Matching: Atrous Matching reduces the referred-pixel set while preserving resolution, addressing the high memory and time cost of high-resolution matching maps.The method targets the computational burden of producing delicate matching maps at stride 4.

4 IMPLEMENTATION DETAILS

The implementation aligns training with inference through balanced cropping and sequential guidance, while CFBI+ extends matching across multiple feature scales. Training uses shared crops, foreground-content restrictions, sequential predictions, and scale-specific embeddings and matching settings.

  • Training strategies: Balanced random-crop uses one window for the first, previous, and current frames, retrying until the first frame contains enough foreground pixels.This addresses foreground-background pixel imbalance and reduces background-biased crops.
  • Training strategies: Sequential training uses the ground-truth previous mask for the first prediction and the latest network prediction for subsequent frames.This makes training more consistent with evaluation, where previous guidance masks come from earlier inference steps.
  • CFBI configuration: CFBI extracts pixel-wise embeddings with 100 channels at stride 4 and downsamples them for multi-local matching to save GPU memory.The embedding feature is reduced to half size before multi-local matching.
  • CFBI+ configuration: CFBI+ extracts three pixel-wise embeddings with 32, 64, and 128 channels at strides 4, 8, and 16, respectively.Its matching windows are configured separately for the three scales.
  • CFBI+ configuration: CFBI+ assigns individual foreground and background biases to each matching scale, with both biases initialized to 0.Group Normalization and Gated Channel Transformation are used to improve stability with small batch sizes.
  • Optimization and data preparation: Training downsamples videos to 480p, uses SGD with momentum 0.9 and bootstrapped cross-entropy on the 15% hardest pixels, plus flipping, scaling, and balanced cropping.The crop size is 465×465 and the scaling range is 1.0 to 1.3 times.

5 EXPERIMENTS

Experiments on YouTube-VOS and DAVIS show that CFBI+ achieves strong accuracy and efficiency without simulated pre-training, while ablations support its foreground-background, multi-scale, and training components.

  • 5.1 Compare with the State-of-the-art Methods: 82.0% average score: CFBI+ outperforms all other methods across every YouTube-VOS evaluation metric without fine-tuning or simulated-data pre-training.A stronger training schedule raises CFBI+2× to 82.8%.
  • 5.2 Ablation Study: 82.0% vs. 81.4%: CFBI+ is more robust than CFBI, while inference improves from 0.29s to 0.25s through multi-scale matching and atrous matching.Atrous matching preserves performance while reducing matching cost across scales.
  • 5.1 Compare with the State-of-the-art Methods: 82.9% J&F: CFBI+ surpasses the challenge Rank 1 result of 81.8% without ensembles, simulated data, or test-time augmentation, at 4 FPS.The improvement mainly comes from unseen categories, supporting generalization beyond seen categories.
  • 5.1 Compare with the State-of-the-art Methods: 82.9% vs. 82.8%: CFBI+ exceeds KMNVOS on DAVIS-2017 without simulated data and has faster multi-object inference at 0.18s versus 0.24s.Evaluation augmentation further raises CFBI+ to 84.5%.
  • 5.1 Compare with the State-of-the-art Methods: 89.9% average score: CFBI+ substantially outperforms FEELVOS at 81.7% on DAVIS-2016 while running faster, although KMNVOS reaches 90.5% using simulated data.On DAVIS-2017, CFBI+ also exceeds KMNVOS by 0.8% under the 600p setting.
  • 5.2 Ablation Study: Removing background embedding lowers performance from 74.9% to 70.9%, while removing pixel-level or instance-level background information yields 73.0% or 72.3%.The ablation supports separately modeling foreground and background features.

6 CONCLUSION

The paper introduces CFBI and CFBI+ for video object segmentation by integrating foreground and background embeddings with pixel-level, instance-level, and multi-scale matching. The resulting framework achieves state-of-the-art accuracy while remaining efficient.

  • 6 CONCLUSION: CFBI integrates contrastive foreground and corresponding background embeddings to address background confusion in video object segmentation.It also combines pixel-level and instance-level embeddings to handle different object scales.
  • 6 CONCLUSION: CFBI+ improves performance through multi-scale matching and improves matching efficiency through atrous matching.The paper presents CFBI and CFBI+ as solid baselines for VOS and related tasks.
Loading 2010.06349v2…