PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Salient Object Detection via Fully Convolutional Networks

Wenguan Wang, Jianbing Shen, Ling Shao

arXiv:1702.00871v3cs.CV

TL;DR

Video salient object detection lacks sufficiently large, densely annotated training data and efficient processing. The paper combines static and dynamic fully convolutional networks with synthetic frame-pair augmentation, avoiding optical flow, and reports high-quality saliency maps on FBMS and DAVIS at 2fps on a GPU.

  • Problem

    Video saliency lacks sufficiently large, densely labeled training data, while existing approaches face costly optical-flow computation.

  • Method

    The model uses static and dynamic fully convolutional networks, explicitly fuses static saliency into dynamic detection, and trains with synthetic frame pairs generated from annotated images.

  • Results

    The model produces more accurate saliency maps than state-of-the-art methods on FBMS, improves performance on DAVIS, and runs at 2fps including all steps on a GPU.

  • Takeaways & Limitations

    The approach provides a fast video saliency detector that captures spatial and temporal cues without optical-flow estimation.

Abstract

from arXiv · show

This paper proposes a deep learning model to efficiently detect salient regions in videos. It addresses two important issues: (1) deep video saliency model training with the absence of sufficiently large and pixel-wise annotated video data, and (2) fast video saliency training and detection. The proposed deep video saliency network consists of two modules, for capturing the spatial and temporal saliency information, respectively. The dynamic saliency model, explicitly incorporating saliency estimates from the static saliency model, directly produces spatiotemporal saliency inference without time-consuming optical flow computation. We further propose a novel data augmentation technique that simulates video training data from existing annotated image datasets, which enables our network to learn diverse saliency information and prevents overfitting with the limited number of training videos. Leveraging our synthetic video data (150K video sequences) and real videos, our deep video saliency model successfully learns both spatial and temporal saliency cues, thus producing accurate spatiotemporal saliency estimate. We advance the state-of-the-art on the DAVIS dataset (MAE of .06) and the FBMS dataset (MAE of .07), and do so with much improved speed (2fps with all steps).

I. INTRODUCTION

The paper applies fully convolutional deep learning to video salient object detection, addressing limited densely labeled video data and costly optical-flow computation. It combines static and dynamic saliency learning with synthetic video augmentation and reports accurate, efficient detection.

  • Existing video datasets are too small, correlated, and costly to annotate for training pixel-wise convolutional networks.
  • Video saliency requires both spatial and temporal scene information, making it more challenging than still-image saliency.
  • The model uses static and dynamic modules to learn spatial and temporal saliency and directly produce spatiotemporal estimates.
  • Optical-flow-free temporal learning from frame pairs makes the proposed model computationally efficient for video saliency detection.
  • 2fps including all steps is achieved on a GPU, while experiments report more accurate saliency maps than state-of-the-art methods on FBMS and improvements on DAVIS.
  • Synthetic video augmentation leverages rich annotated image datasets to encode diverse static and dynamic saliency information while limiting overfitting.

II. RELATED WORK

Related work covers saliency detection and deep learning for dynamic scenes. Video saliency is harder than image saliency because it must handle temporal and motion information, while optical-flow-based approaches incur high computational costs.

  • A. Saliency Detection: Saliency research includes visual attention prediction and salient object detection, with the latter uniformly highlighting salient regions for vision applications.
  • A. Saliency Detection: Image saliency methods commonly use bottom-up contrast and other prior knowledge to identify visually distinctive regions.
  • A. Saliency Detection: Video saliency is more challenging than still-image saliency because detecting and using temporal and motion information is complicated.
  • A. Saliency Detection: Existing video saliency approaches extend static models with temporal information, but their applicability is severely limited by high computational costs.
  • A. Saliency Detection: Deep video methods using multi-frame dense optical flow incur heavy computational burdens, motivating alternatives that avoid optical-flow computation.

B. Deep Learning Models in Dynamic Scenes

The paper positions CNN-based video saliency within dynamic-scene vision and proposes offline training from real and synthetic videos to learn static and dynamic cues efficiently.

  • Related deep models: Prior dynamic-scene networks often process frames with temporal post-processing or use optical flow, creating a trade-off between temporal modeling and computational cost.Some approaches omit learned temporal information, while two-stream methods rely on multi-frame optical flow and incur heavy computation.
  • Training scheme: Synthetic and real labelled video data support fully convolutional training for learning both static and dynamic saliency features.The training procedure leverages large image datasets through synthetic video generation and is performed offline.
  • Efficiency: The model directly learns relationships between adjacent frames, avoiding time-consuming motion computation and improving efficiency over optical-flow-based methods.The approach focuses on short-term frame-pair analysis rather than long-term information from multiple adjacent frames.
  • Proposed model: The proposed model uses two modules: a static saliency network for single frames and a dynamic network for frame pairs, with static estimates serving as prior information.The dynamic module directly produces final spatiotemporal saliency maps.

B. Deep Networks for Static Saliency

The static saliency branch is a fully convolutional, pixel-wise predictor that transforms an input image into a same-sized probability map through convolution, upsampling, and sigmoid output.

  • Network architecture: The static network extracts multidimensional feature representations with convolutional layers and restores spatial resolution using multilayer deconvolution networks.Downsampling makes convolutional features coarse, so deconvolution layers upsample them for pixel-wise prediction.
  • Feature extraction: Each convolutional layer applies a trainable kernel and bias, followed by point-wise nonlinearities and often nonlinear downsampling to improve feature representations.The resulting features tolerate small spatial variations in the input.
  • Network architecture: The network maps upsampled feature maps to a saliency prediction through a 1 × 1 convolution and sigmoid activation, producing values between 0 and 1.The fully convolutional design preserves spatial information and supports input images of arbitrary sizes.
  • Optimization: Training minimizes a weighted cross-entropy loss between the predicted probability map and ground-truth saliency mask.The weighting addresses imbalance between salient and non-salient pixels.
  • Optimization: The convolutional layers are initialized from the first five VGGNet blocks trained on 1.3 million ImageNet images, while remaining layers are randomly initialized.The network is trained end-to-end with stochastic gradient descent.

C. Deep Networks for Dynamic Saliency

The dynamic saliency branch combines adjacent frames with static saliency priors in an FCN, directly producing spatiotemporal estimates while avoiding optical-flow inputs.

  • Dynamic network: The dynamic network uses an FCN architecture similar to the static branch and jointly learns dynamic saliency with static saliency results.Its output is a direct spatiotemporal saliency estimate.
  • Training and inference: During training, the model uses synthetic and real videos, feeding frame pairs and the ground-truth mask for frame I_t to learn dynamic saliency.The static prior is generated by the static saliency network and contributes to the final estimate.
  • Dynamic network: Successive frames (I_t, I_t+1) and the static saliency prior are concatenated channel-wise into a h × w × 7 tensor before dynamic processing.The first convolutional layer is modified to accept the concatenated input.
  • Design comparison: Unlike two-stream architectures, the model merges static-network output into the dynamic network rather than requiring a separate fusion network.This explicitly integrates spatial and temporal saliency within the dynamic model.
  • Synthetic data: Synthetic video generation warps an image and its saliency mask according to a smoothed optical-flow field that simulates real motion patterns.The field is initialized from varied motion characteristics and refined through the paper’s optimization procedure.
  • Efficiency: Using two adjacent frames instead of optical-flow images increases computational efficiency.The model directly generates spatiotemporal results after receiving the frame pair and static prior.

IV. SYNTHETIC VIDEO DATA GENERATION

The method synthesizes short video frame pairs from annotated images by modeling superpixel motion, smoothing it while preserving foreground–background differences, and generating transformed frames and labels.

  • The approach addresses overfitting caused by insufficiently diverse and small annotated video datasets.
  • It generates two-frame video samples from annotated image–mask pairs, producing diverse motion patterns, deformations, and smooth transformations.
  • Optical flow is initialized on foreground and background superpixels, with 10% of background regions receiving random vertical motion in [−d, d], where d = h/10.
  • An energy function smooths and propagates superpixel motion while preserving motion differences between foreground and background regions.
  • The process efficiently generates synthesized frame pairs, optical flow, and pixel-wise annotations from large image saliency datasets.Using MSRA10K and DUT-OMRON, the method generates more than 150K simulated videos.

V. EXPERIMENTAL RESULTS

The experiments evaluate the proposed approach on two large video datasets, comparing its performance, component importance, and computational load.

  • The evaluation covers exhaustive comparisons across two datasets containing 80 videos in total.
  • The experimental protocol includes performance comparisons, component analysis, and computational-load assessment.

1) Datasets:

The experiments use FBMS and DAVIS for evaluation, image saliency datasets for training, and comparisons with multiple alternative saliency methods.

  • Datasets: FBMS contains 59 natural video sequences with appearance variation, shape deformation, and camera motion; the study supplies more precise salient-object annotations.
  • Datasets: Training uses MSRA10K with 10K images and DUT-OMRON with 5172 cluttered multi-object images.
  • Implementation: Because FBMS provides only 4–6 labeled frames per sequence, the study adds approximately 500 annotations.
  • Implementation: The aggregate training set combines more than 150K simulated video slits with approximately 3K real video frame pairs.
  • Performance Comparison: The method is compared with six named alternatives spanning image- and video-saliency approaches.

1) Qualitative Results:

The proposed method is evaluated qualitatively and quantitatively using saliency maps, precision-recall curves, F-scores, and MAE on FBMS and DAVIS.

  • Qualitative Results: Qualitatively, the method captures salient foreground objects in blurred backgrounds, varied motion patterns, and large shape deformations.
  • Quantitative Results: The evaluation reports precision-recall curves, F-measure, and mean absolute error as complementary performance measures.
  • Quantitative Results: MAE measures the average per-pixel difference between normalized estimated saliency probabilities and ground truth.
  • Quantitative Results: The proposed method significantly outperforms state-of-the-art methods on both FBMS and DAVIS in precision-recall performance.
  • Quantitative Results: It achieves better F-scores and the lowest MAE among all compared methods.

C. Validation of the Proposed Method

The validation studies evaluate component contributions, training strategies, and data quantity on FBMS and DAVIS using precision-recall curves and MAE. Results show that combining static and dynamic cues helps, synthetic data are effective but do not replace real videos, and more training data improves performance.

  • Evaluation setup: The study evaluates FBMS and DAVIS with precision-recall curves and MAE, including component variants and training strategies.
  • Component analysis: Static-only saliency increased MAE from 7.65 to 8.19 on FBMS and from 6.36 to 7.17 on DAVIS.The passage attributes the decrease to missing dynamic saliency information.
  • Component analysis: Fusing static and dynamic models improves both components, while static saliency as a prior makes dynamic-model training easier and predictions more accurate.
  • Training strategies: Using only synthetic video data increased MAE from 7.65 to 9.27 on FBMS and from 6.36 to 7.53 on DAVIS.The small decrease in performance supports the augmentation strategy but indicates synthetic data should not completely replace real video.
  • Training strategies: Reducing the amount of training data decreased performance, indicating that the deep-learning model is data-driven.Conversely, the passage reports that increasing training data improves performance.

D. Runtime Analysis

The runtime analysis examines the proposed model against prior video saliency methods and emphasizes its reduced computational burden. By avoiding optical flow and other pre-computed information, the method achieves fast 480p processing and 2fps GPU operation.

  • Evaluation conditions: Runtime comparisons use an Intel Xeon E7 CPU, 64 GB memory, and an Nvidia Geforce TITAN X GPU, excluding I/O and parallel image processing.
  • Runtime comparison: 0.47 seconds is sufficient for the method to compute a 480p saliency mask, roughly an order of magnitude faster than traditional video saliency methods.
  • Computational design: The method avoids optical flow, edge maps, and other pre-computed information, reducing the computational burdens associated with previous approaches.
  • Computational design: The model uses static and dynamic saliency modules to capture spatial and temporal statistics, with static estimates incorporated into dynamic detection.
  • Runtime comparison: 2fps on a GPU is the reported processing frame rate for the saliency model.
Loading 1702.00871v3…