Source-linked AI summary
CNN in MRF: Video Object Segmentation via Inference in A CNN-Based Higher-Order Spatio-Temporal MRF
Linchao Bao, Baoyuan Wu, Wei Liu
TL;DR
The paper addresses semi-supervised video object segmentation from a first-frame mask, where combining strong CNN appearance modeling with temporal reasoning remains difficult. It embeds CNN-based higher-order spatial potentials in a spatio-temporal MRF and performs approximate inference by alternating temporal fusion and CNN refinement. The method outperforms winning DAVIS 2017 Challenge entries without model ensembling or dedicated object detectors.
Problem
Semi-supervised video object segmentation requires propagating class-agnostic object masks from the first frame, but CNN and graph-based methods provide incomplete combinations of appearance and temporal modeling.
Method
The paper uses a pixel-level spatio-temporal MRF with optical-flow temporal dependencies, CNN-encoded higher-order spatial potentials, and alternating temporal fusion and feed-forward mask refinement.
Results
The method outperforms the winning entries of the DAVIS 2017 Challenge without model ensembling or dedicated object detectors.
Takeaways & Limitations
Embedding a feed-forward CNN inside MRF inference combines object appearance and shape representations with spatio-temporal connections for class-agnostic video segmentation.
Takeaways & Limitations
Inference is very difficult because the CNN-based energy creates very high-order dependencies, so the paper uses an efficient approximate algorithm.
Abstract
from arXiv · showhide
This paper addresses the problem of video object segmentation, where the initial object mask is given in the first frame of an input video. We propose a novel spatio-temporal Markov Random Field (MRF) model defined over pixels to handle this problem. Unlike conventional MRF models, the spatial dependencies among pixels in our model are encoded by a Convolutional Neural Network (CNN). Specifically, for a given object, the probability of a labeling to a set of spatially neighboring pixels can be predicted by a CNN trained for this specific object. As a result, higher-order, richer dependencies among pixels in the set can be implicitly modeled by the CNN. With temporal dependencies established by optical flow, the resulting MRF model combines both spatial and temporal cues for tackling video object segmentation. However, performing inference in the MRF model is very difficult due to the very high-order dependencies. To this end, we propose a novel CNN-embedded algorithm to perform approximate inference in the MRF. This algorithm proceeds by alternating between a temporal fusion step and a feed-forward CNN step. When initialized with an appearance-based one-shot segmentation CNN, our model outperforms the winning entries of the DAVIS 2017 Challenge, without resorting to model ensembling or any dedicated detectors.
1. Introduction
The paper targets semi-supervised, class-agnostic video object segmentation and addresses a gap between CNN-based appearance modeling and graph-based temporal reasoning. It proposes a CNN-encoded higher-order spatio-temporal MRF with an alternating inference algorithm to combine these cues.
- 1. Introduction: Semi-supervised video object segmentation propagates class-agnostic object masks from a manually annotated first frame through subsequent video frames.The task extracts pixel-level masks and supports applications including video editing, summarization, and action recognition.
- 1. Introduction: CNN-based methods provide strong representations but often process frames independently or use simple temporal heuristics, whereas graph models cannot exploit neural-network representations.The paper identifies this separation as a gap motivating a more principled combination of appearance and temporal information.
- 1. Introduction: The proposed spatio-temporal MRF assigns pixel labels, uses optical flow for temporal dependencies, and encodes spatial dependencies with CNN-based higher-order potentials.The CNN evaluates masks holistically, enabling richer pixel interactions than conventional pairwise spatial potentials.
- 1. Introduction: The paper presents the CNN-encoded MRF and alternating inference procedure as its main contributions for video object segmentation.The approach is initialized with an appearance-based one-shot segmentation CNN and is evaluated on public benchmarks.
- 1. Introduction: The inference algorithm alternates temporal fusion with a feed-forward CNN mask-refinement step to progressively improve segmentation results.This design combines temporal information flow with object-specific appearance and shape information.
2. Model
The model defines a pixel-level spatio-temporal MRF for semi-supervised video object segmentation, combining optical-flow temporal links with CNN-encoded higher-order spatial dependencies. Its spatial energy evaluates whole-frame masks through a mask-refinement CNN, while approximate inference addresses the resulting difficult optimization.
- 2.1. Notations & Preliminaries: Each pixel is a binary random variable in a video-wide field, and MAP inference minimizes data-conditioned total energy over label assignments.The total energy combines unary, temporal, and spatial terms.
- 2.2. Model Structures & Energies: The unary term uses each pixel’s negative log-likelihood, while temporal links connect pixels across frames using reliable semi-dense optical flow.A forward-backward consistency check filters unreliable motion vectors, and the temporal term encourages consistency when connections are confident.
- 2.2. Model Structures & Energies: Temporal connections use neighboring frames and can extend to k-step links; the model uses k ⩽2, giving each pixel at most four temporal neighbors before invalid links are removed.The listed links are between frames t and t−2, t−1, t+1, and t+2.
- 2.2. Model Structures & Energies: All pixels within a frame form one spatial clique, so each pixel’s labeling depends on the other pixels in that frame.This higher-order structure is intended to assess a complete mask rather than independent local pixel relations.
- 2.2. Model Structures & Energies: A mask-refinement CNN maps a candidate mask to a refined mask, and the spatial energy favors masks whose CNN refinement remains similar to the input.The function f(x_c) measures the squared difference between the candidate mask and its CNN-refined output.
- 2.2. Model Structures & Energies: CNN-based spatial energies offer greater expressive power than conventional pairwise or limited higher-order energies, but they make MRF inference very difficult.The paper therefore introduces an efficient approximate inference algorithm for the CNN-based energy.
3. Inference
The method makes higher-order CNN-based MRF inference tractable by decoupling temporal and spatial energies and alternating temporal fusion with CNN mask refinement. The algorithm uses approximate updates and experimentally converges after a few iterations.
- Inference challenge: The higher-order CNN-based energy makes exact and standard approximate MAP inference computationally infeasible.Evaluating the MRF energy would require a feed-forward CNN pass for every video frame.
- Alternating optimization: An auxiliary variable y decouples temporal and spatial energies, producing an approximate objective with alternating x and y updates.The penalty parameter β keeps y close to x while the updates separate temporal fusion from mask refinement.
- Temporal fusion: The x-update performs approximate temporal fusion by iteratively updating individual variables with Iterated Conditional Modes.Only a fixed number L of ICM iterations is performed for efficiency.
- Convergence: 99% of more than 3000 DAVIS 2017 validation frames showed a non-increasing mask-refinement objective when θs = β, and the full algorithm converged after a few iterations.The procedure alternates temporal fusion and mask refinement, then binarizes the final output masks.
- Mask refinement: The y-update refines masks frame by frame using the CNN operator gCNN(·), which takes an RGB image and coarse mask as input and outputs a refined mask.The CNN is trained offline and then fine-tuned using the first-frame ground-truth mask for the given video.
4. Experiments
Experiments show that alternating temporal fusion with CNN-based mask refinement improves segmentation across challenging benchmarks, while each component alone is limited. The method achieves strong results without dedicated object detectors or heavy engineering techniques.
- Implementation Details: The implementation uses OSVOS to initialize labeling and pixel likelihoods for all frames, with Gaussian weighting to reduce false positives from similar objects.The weighting is centered at the target location predicted by a linear motion model.
- Ablation Study: Temporal fusion alone degrades segmentation because it ignores rich spatial dependencies and can propagate erroneous labels across neighboring frames.Mask refinement alone improves the baseline but its gain stalls around 5%.
- Ablation Study: With both temporal fusion and mask refinement enabled, the method improves the baseline by up to 11%.Temporal fusion recovers missing segments but introduces coarse false positives, which mask refinement can correct; the experiments use K = 3 iterations.
- Results: On DAVIS 2017 test-dev, the algorithm outperforms the challenge-winning entries without model ensembling, multi-scale training/testing, or dedicated object detectors.The benchmark includes heavy occlusions, appearance changes, shape deformation, and diverse object scales.
- Results: Qualitative DAVIS 2017 examples cover the carousel, girl-dog, and salsa sequences, with multiple objects highlighted in different colors.The first column contains the initial ground-truth mask, followed by results for subsequent frames.
- Results: The method achieves state-of-the-art performance on DAVIS 2016, Youtube-Objects, and SegTrack v2 using the same parameter settings.These legacy datasets are less challenging and have more saturated scores than DAVIS 2017.
5. Conclusions
The paper proposes a spatio-temporal MRF for video object segmentation and an inference algorithm alternating temporal fusion with mask-refinement CNN feed-forward passes.
- The proposed model performs video object segmentation using a novel spatio-temporal MRF.
- Its inference algorithm alternates temporal fusion with a mask refinement feed-forward CNN to progressively infer segmentation results.
- Extensive experiments on challenging datasets demonstrate the effectiveness of the proposed algorithm.
- The method embeds a feed-forward CNN pass inside MRF inference, exploring a new direction for combining MRFs and CNNs.