PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Propagation Networks

Varun Jampani, Raghudeep Gadde, Peter V. Gehler

arXiv:1612.05478v3cs.CV

TL;DR

Video propagation requires transferring structured information across frames despite motion, computational cost, and limited suitability of static CNN receptive fields. The paper proposes online Video Propagation Networks that combine temporal bilateral filtering with spatial refinement, and reports better, faster performance across video propagation tasks.

  • Problem

    Propagating structured information across video frames is difficult because motion creates pixel-association challenges and video processing is computationally demanding.

  • Method

    VPNs combine a temporal bilateral network for dense, video-adaptive propagation with a spatial CNN that refines predictions for the current frame.

  • Results

    VPNs outperformed published approaches on video object segmentation and semantic label propagation while being faster.

  • Takeaways & Limitations

    The architecture provides generic online propagation for discrete labels and continuous values, including semantic information and color.

  • Takeaways & Limitations

    The video object segmentation experiments omit optical-flow information to remain comparable with a fast state-of-the-art technique.

Abstract

from arXiv · show

We propose a technique that propagates information forward through video data. The method is conceptually simple and can be applied to tasks that require the propagation of structured information, such as semantic labels, based on video content. We propose a 'Video Propagation Network' that processes video frames in an adaptive manner. The model is applied online: it propagates information forward without the need to access future frames. In particular we combine two components, a temporal bilateral network for dense and video adaptive filtering, followed by a spatial network to refine features and increased flexibility. We present experiments on video object segmentation and semantic video segmentation and show increased performance comparing to the best previous task-specific methods, while having favorable runtime. Additionally we demonstrate our approach on an example regression task of color propagation in a grayscale video.

1. Introduction

The paper introduces Video Propagation Networks (VPNs), a generic online architecture for propagating structured information across video frames. VPNs combine video-adaptive bilateral filtering with spatial refinement to support multiple propagation tasks with favorable runtime.

  • Motivation: Scene and camera motion create difficult pixel associations, while video computation and per-frame processing make standard CNN approaches challenging.These difficulties have contributed to the use of slow optimization-based methods for video propagation.
  • Problem and contribution: VPNs propagate structured information across video frames, including semantic labels and colors, using only current and previous frames.The method is designed for online propagation rather than access to future frames.
  • Architecture: A temporal bilateral network densely filters pixels from current and previous frames, then a spatial CNN refines and predicts the present frame.The bilateral component supplies video-adaptive temporal information, while the spatial network adds refinement and flexibility.
  • Properties: VPNs are end-to-end trainable, long-range, image-adaptive, and applicable to both discrete and continuous information.The stated examples include semantic labels and color information.
  • Results: VPNs outperformed published approaches on video object segmentation and semantic label propagation while being faster.The paper also reports that VPNs can be integrated into sequential per-frame approaches with a separate small fine-tuning step.

2. Related Work

Prior video propagation methods are dominated by optimization and filtering approaches, with work spanning graph models, correspondence estimation, and temporal semantic segmentation. VPN builds on learnable, image-adaptive bilateral filtering to provide a generic propagation architecture.

  • General propagation techniques: Optimization-based propagation commonly formulates video information transfer as energy minimization on graphs, but large or complex video connectivity can make it slow.Fully connected CRFs provide dense long-range connections while retaining fast inference.
  • General propagation techniques: Filtering techniques propagate information with image or video filters and generally provide faster runtimes than optimization-based methods.Bilateral filtering is highlighted as a popular long-range propagation filter.
  • Relation to VPN: VPN treats learnable bilateral filtering as the core of a video propagation method, extending image-adaptive filtering to video processing.The paper positions this filtering formulation as the basis for its approach.
  • Video object segmentation: Video object segmentation work includes semi-supervised methods that propagate a first-frame foreground mask and unsupervised methods using motion or saliency priors.Semi-supervised approaches have used graph cuts, clustering, optical flow, and pixel correspondences.
  • Semantic video segmentation: Semantic video segmentation methods have used geometric or motion features, graphical models, temporal links, and propagated CNN predictions.Recent approaches also optimize feature spaces for fully connected CRFs.

3. Bilateral Filtering

Bilateral filtering is an edge-preserving, content-adaptive operation whose weights depend on feature relationships between pixels. Efficient lattice-based implementations make this operation practical, while VPN uses its adaptive connectivity for moving video scenes.

  • Video adaptation: VPN adapts bilateral filtering to videos so its connectivity responds to scene and camera motion that fixed-size CNN receptive fields cannot adequately represent.The paper introduces a video-adaptive filtering component for this purpose.
  • Bilateral filtering: Bilateral filtering can be represented as matrix-vector multiplication, with filter weights determined by features at pairs of input pixels.The feature representation controls how the filter adapts to image content.
  • Bilateral filtering: Using positional features produces spatial Gaussian filtering, while combining position and color features enables edge-preserving smoothing.The cited formulation uses feature-dependent weights to adapt filtering to content.
  • Efficient computation: Because filter values vary across pixel pairs and videos contain many pixels, naively constructing the filter matrix is computationally prohibitive.Fast algorithms avoid explicitly building the full matrix.
  • Efficient computation: The bilateral-grid formulation expresses filtering as splatting, convolution in bilateral space, and slicing back to image pixels.The bilateral space has the same dimensionality as the feature representation.
  • Efficient computation: The permutohedral lattice avoids the exponential grid-point cost of regular high-dimensional convolution and yields filtering that scales linearly with feature dimension.This structure supports fast execution of the bilateral operation.

4. Video Propagation Networks

Video Propagation Networks propagate information online from previous frames using video-adaptive bilateral filtering followed by spatial refinement. The architecture is learnable, supports multiple information types, and adapts its connectivity to scene and object motion.

  • Video Propagation Networks: VPNs learn an online mapping from current and previous video frames and propagated outputs to the present-frame prediction.The formulation predicts v_t after observing the video only up to frame t.
  • Video Propagation Networks: The method combines a temporal bilateral network for dense, video-adaptive filtering with a spatial CNN that refines the current-frame output.The bilateral network connects pixels across current and previous frames, while the spatial network refines the resulting features.
  • Bilateral Network (BNN): Bilateral propagation uses previous-frame values, video features, lattice splatting, learned filtering, and slicing into the current frame.A bilateral convolution layer consists of splatting, convolving, and slicing; the filter bank is learned by back-propagation, with 1 × 1 filters selected for speed.
  • Bilateral Network (BNN): The architecture adapts its connectivity to video content and motion through image features, including spatial, color, and time coordinates.Optical flow can warp previous-frame position features toward the current frame, while the reported experiments use time as an additional feature.
  • Bilateral Network (BNN): Permutohedral sampling can provide big computational gains with a minor performance drop, while learnable filter banks increase flexibility beyond fixed Gaussian filtering.The method uses sampled points or superpixels for filtering and learns the bilateral filter bank through back-propagation.
  • Bilateral Network (BNN): The BNN stacks bilateral convolution layers with ReLU nonlinearities, concatenation, and a spatial projection before producing refined predictions.The described architecture uses two initial 32-filter bilateral layers, a second filter-bank stage, and a 1 × 1 spatial filter mapping back to the output dimension.

5. Experiments

Experiments evaluate VPN on video object segmentation and semantic video segmentation, showing improved performance over prior methods while retaining favorable runtime. The experiments also examine staged refinement, adaptive sampling, DeepLab integration, and qualitative robustness.

  • Experimental setup: VPN was evaluated on foreground-mask propagation, semantic label propagation, and color propagation, using Adam with task-specific multinomial-logistic or Euclidean losses.The implementation used Caffe, a fixed learning rate of 0.001, and selected models by minimum validation loss.
  • Video object segmentation: DAVIS evaluation used 5-fold cross-validation on 50 videos, with 35 training, 5 validation, and 10 testing videos per fold.The evaluation used IoU, contour accuracy, and temporal instability metrics.
  • Video object segmentation: Sampling 25% of the roughly 2 million points reached a plateau in BNN-Identity IoU; superpixel sampling reduced IoU by 0.5 while reducing input points tenfold.The final setup used 12,000 SLIC superpixels per frame and mask probabilities from the previous nine frames.
  • Video object segmentation: VPN improved consistently across all five DAVIS folds as additional stages were added, outperforming BVS on all performance measures with comparable runtime.VPN was marginally better than OFL while being at least 80× faster and achieved similar performance without optical flow; unlike BVS, it processes online using previous frames.
  • Video object segmentation: VPN-DeepLab achieved 75.0 IoU with a 0.63s runtime per frame, combining bilateral propagation with DeepLab predictions and end-to-end spatial refinement.This model was reported as a significant improvement over published results and one of the fastest techniques.
  • Semantic video segmentation: On CamVid semantic segmentation, full VPN training improved CNN-1 performance by over 1.2 IoU and ran at least 25× faster than FSO-CRF.Replacing bilateral filters with spatial filters improved CNN-1 by only 0.3 IoU, while VPN-Flow further improved CNN-2.

6. Conclusion

The paper presents VPN as a fast, scalable, generic approach for propagating information across video frames. Experiments across diverse tasks reportedly outperform current task-specific methods, while future work targets better learned bilateral features and scales.

  • VPN propagates information from previous frames to the present frame using long-range, video-adaptive bilateral processing followed by spatial refinement.
  • Experiments on diverse tasks show that generic VPNs outperformed current state-of-the-art task-specific methods.
  • The showcased architecture is intentionally simple; more filters or deeper layers could improve performance depending on the problem and available data.
  • Learning feature scales and finding optimal fast-computing bilateral video features are identified as important future research directions.

A. Parameters and Additional Results

The supplementary material documents experiment protocols and visual results for video object segmentation, semantic video segmentation, and video color propagation. The figures compare inputs and ground truth with predictions from VPN variants and prior methods.

  • Parameters: The experiment protocols specify bilateral feature types, feature scales, temporal decay, and the number of input frames.
  • Additional Results: Video object segmentation figures compare ground-truth masks with predictions from BVS, OFL, VPN, and VPN-DLab across example frames.
  • Additional Results: Semantic video segmentation figures show input frames, ground-truth segmentation, CNN predictions, and VPN-Flow predictions.
  • Additional Results: Video color propagation figures compare grayscale video inputs and ground-truth colors with predictions from Levin et al. and VPN-Stage1.
Loading 1612.05478v3…