PaperlayerView exact revision on arXiv

Source-linked AI summary

A Benchmark Dataset and Saliency-guided Stacked Autoencoders for Video-based Salient Object Detection

Jia Li, Changqun Xia, Xiaowu Chen

arXiv:1611.00135v2cs.CV

TL;DR

Video-based salient object detection lacks large datasets with unambiguous annotations. The paper introduces VOS, combining object masks and multi-subject eye tracking, and proposes saliency-guided stacked autoencoders; the resulting benchmark shows the dataset is challenging and the unsupervised approach outperforms 30 models.

  • Problem

    Video-based SOD is less explored because large-scale video datasets with unambiguous salient-object definitions and annotations are lacking.

  • Method

    The paper constructs VOS from object and region masks plus eye-tracking data, then uses saliency-guided stacked autoencoders with multi-level spatiotemporal cues.

  • Results

    The proposed unsupervised approach outperforms 30 image-based and video-based models on VOS.

  • Takeaways & Limitations

    VOS provides a challenging benchmark for video-based SOD and supports studying its differences and correlations with image-based SOD.

Abstract

from arXiv · show

Image-based salient object detection (SOD) has been extensively studied in the past decades. However, video-based SOD is much less explored since there lack large-scale video datasets within which salient objects are unambiguously defined and annotated. Toward this end, this paper proposes a video-based SOD dataset that consists of 200 videos (64 minutes). In constructing the dataset, we manually annotate all objects and regions over 7,650 uniformly sampled keyframes and collect the eye-tracking data of 23 subjects that free-view all videos. From the user data, we find salient objects in video can be defined as objects that consistently pop-out throughout the video, and objects with such attributes can be unambiguously annotated by combining manually annotated object/region masks with eye-tracking data of multiple subjects. To the best of our knowledge, it is currently the largest dataset for video-based salient object detection. Based on this dataset, this paper proposes an unsupervised baseline approach for video-based SOD by using saliency-guided stacked autoencoders. In the proposed approach, multiple spatiotemporal saliency cues are first extracted at pixel, superpixel and object levels. With these saliency cues, stacked autoencoders are unsupervisedly constructed which automatically infer a saliency score for each pixel by progressively encoding the high-dimensional saliency cues gathered from the pixel and its spatiotemporal neighbors. Experimental results show that the proposed unsupervised approach outperforms 30 state-of-the-art models on the proposed dataset, including 19 image-based & classic (unsupervised or non-deep learning), 6 image-based & deep learning, and 5 video-based & unsupervised. Moreover, benchmarking results show that the proposed dataset is very challenging and has the potential to boost the development of video-based SOD.

I. INTRODUCTION

Video-based SOD lacks large datasets with unambiguous salient-object annotations, limiting model development and fair benchmarking. The paper introduces VOS, an eye-tracking- and mask-based dataset, plus an unsupervised baseline and comprehensive benchmark.

  • I. INTRODUCTION: VOS addresses limited video-dataset scale and ambiguous salient-object annotations by combining manually annotated objects with multi-subject eye-tracking data.The dataset contains 200 videos and produces binary salient-object masks for 7,467 keyframes after filtering.
  • I. INTRODUCTION: The proposed saliency-guided stacked autoencoders use pixel-, superpixel-, and object-level spatiotemporal cues to infer pixel saliency from local temporal neighborhoods.The model is presented as an unsupervised baseline for video-based SOD.
  • I. INTRODUCTION: VOS is organized to support comparisons across video difficulty, with VOS-E containing 97 easy videos and VOS-N containing 103 normal videos.The subsets differ in foreground, background, and motion complexity.
  • I. INTRODUCTION: The paper benchmarks the proposed approach and 30 image-based and video-based models on VOS to expose challenges in video-based SOD.The benchmark includes image-based classic, image-based deep-learning, and video-based unsupervised models.
  • I. INTRODUCTION: Existing video datasets are smaller, cover limited scenarios, or use subjective and ambiguous foreground or salient-object annotations.The paper identifies limited video numbers, annotation bias, and mismatches between foreground, moving, and salient objects as key shortcomings.

B. Models

Prior video-based SOD models are less developed and less quantitatively benchmarked than image-based SOD models. Existing approaches use spatiotemporal cues, fixation priors, motion or appearance models, but require broader and more general evaluation.

  • B. Models: Image-based SOD models include bottom-up, learning-based, deep, and recurrent architectures that progressively refine salient-region predictions.The reviewed examples include stacked denoising autoencoders, hierarchical recurrent CNNs, and iterative refinement networks.
  • B. Models: Video-based SOD is less explored than image-based SOD because large-scale video datasets are scarce, leading many early approaches to provide mainly qualitative comparisons.Only a few works report quantitative results, often on small datasets with roughly annotated salient objects.
  • B. Models: Single-video methods model temporal coherence using regional dynamic contrast, color, texture, and motion, but often assume one salient object or rely on restrictive visual distinctions.These assumptions may not hold in complex real-world scenes.
  • B. Models: Other approaches extend image co-segmentation to videos by learning global appearance models or detecting shared foreground objects across video sets.These methods target shared foreground structure rather than the paper’s broader salient-object formulation.

III. A LARGE-SCALE DATASET FOR VIDEO-BASED SOD

VOS is constructed from realistic indoor and outdoor videos, dense object and region masks on uniformly sampled keyframes, and eye-tracking data from multiple viewers. Salient objects are selected by combining fixation density with object annotations to reduce ambiguity.

  • III. A LARGE-SCALE DATASET FOR VIDEO-BASED SOD: VOS contains 200 realistic indoor and outdoor videos lasting 64 minutes, divided into 97 easy videos and 103 normal videos.VOS-E emphasizes obvious foreground objects and slow camera motion, whereas VOS-N contains complex or highly dynamic foregrounds and cluttered backgrounds.
  • III. A LARGE-SCALE DATASET FOR VIDEO-BASED SOD: Combining object masks and human fixations is intended to reduce ambiguity and subjective bias in defining salient objects in complex scenes.The construction explicitly treats both annotation types as complementary user data.
  • III. A LARGE-SCALE DATASET FOR VIDEO-BASED SOD: Four subjects annotate all objects and regions on 7,650 uniformly sampled keyframes, preserving object identities across each video.The annotation process records accurate object boundaries and ignores regions smaller than 16 pixels.
  • III. A LARGE-SCALE DATASET FOR VIDEO-BASED SOD: Twenty-three additional subjects free-view all videos while a 500Hz eye tracker records fixation positions and timestamps.The mask annotators and eye-tracking participants are distinct groups, and viewing procedures are designed to reduce fatigue.

C. Definition and Annotation of Salient Objects in Video

Frame-level fixations are unreliable for defining salient objects in complex videos because attention is sparse, inaccurate, distracted, and sometimes recorded only on background frames. The paper therefore defines video saliency over entire videos using consistently high fixation density, then combines this definition with object masks to generate salient-object annotations.

  • Limitations of frame-level fixation: Multiple-subject fixations reduce subjective bias but remain insufficient for directly annotating salient objects separately on each video frame.Different candidate objects and annotator biases make direct manual annotation ambiguous in complex videos.
  • Limitations of frame-level fixation: Frame-level fixation annotation fails because viewing time is short, fixations can miss small moving objects, attention can shift to surprising backgrounds, and some frames contain only background.These failure modes prevent reliable per-frame distinction between salient objects and distractors.
  • Definition of video saliency: Video saliency is defined by objects that consistently receive the highest fixation densities throughout the entire video, rather than by frame-level fixation counts.This video-level definition reduces errors from small objects, background regions, and transient attention shifts.
  • Annotation procedure: The annotation process combines manually annotated object and region masks with fixation data from multiple subjects to identify salient objects.The supplied figures illustrate both the object-mask annotations and the fixation patterns motivating the video-level definition.
  • Annotation procedure: Object fixation density is computed using spatial and temporal weighting, so fixations farther from an object in space or time contribute less.The weighting parameters are empirically set using σs and σt, with spatial scale based on video size and temporal scale of 0.1s.
  • Annotation procedure: Objects with saliency scores above an empirical threshold of 50 are selected, or the highest-scoring object is selected when no score reaches 50.The resulting salient objects are represented as binary masks at keyframes.

E. Dataset Statistics

VOS is center-biased, with salient-object distributions resembling challenging image datasets while VOS-E offers a balanced range of single-object sizes for benchmarking.

  • VOS and its subsets are center-biased, slightly more strongly than ASD, ECSSD, and DUT-O.The paper attributes this pattern to photographers’ tendency to place salient targets near the view center.
  • VOS has salient-object counts and areas similar to DUT-O, reflecting realistic and challenging scenarios.
  • VOS-E contains almost exclusively one salient object per keyframe, with sizes distributed across Small (31.1%), Medium (30.1%), Large (20.6%), and Very Large (18.3%).These distributions support using VOS-E as a baseline benchmark dataset.

IV. A BASELINE MODEL FOR VIDEO-BASED SOD WITH SALIENCY-GUIDED STACKED AUTOENCODERS

The baseline combines pixel-, superpixel-, and object-level spatiotemporal saliency cues, then uses saliency-guided stacked autoencoders to estimate video saliency.

  • B. Extracting Multi-scale Saliency Cues: The approach extracts spatiotemporal saliency cues at pixel, superpixel, and object levels before combining them in the baseline framework.Frames are represented in multiple color spaces, with object proposals and motion information also extracted.
  • B. Extracting Multi-scale Saliency Cues: Pixel-based saliency uses minimum barrier distances, backgroundness enhancement, and morphological smoothing, but remains sensitive to noise.
  • B. Extracting Multi-scale Saliency Cues: Superpixel-based saliency detects large salient objects as wholes, while object-based saliency can include nearby background regions.
  • B. Extracting Multi-scale Saliency Cues: Object-based saliency uses object proposals and fixation-density maps in Lab, HSV, and XYT spaces to assign proposal-derived saliency.

C. Learning Stacked Autoencoders

The learning stage trains stacked autoencoders on local spatiotemporal saliency cues, using smoothness and consistency assumptions to derive pixel-level scores and refine maps.

  • The autoencoders use saliency cues from each pixel, eight spatial neighbors, and one flow-defined temporal neighbor to model spatial smoothness and temporal consistency.Each pixel is represented by 30 saliency cues.
  • Training is unsupervised: sampled saliency-cue vectors are passed through progressively trained encoding and decoding layers without ground-truth data.The model uses 500,000 randomly selected VOS pixels and logistic-sigmoid transfer functions.
  • The final encoding layer has one neuron, whose output is correlated with reconstructed local saliency cues to obtain pixel saliency scores.The implementation uses four encoding layers with 15, 7, 3, and 1 neurons.
  • Post-processing applies temporal Gaussian smoothing, foreground/background contrast enhancement, and threshold-based cleanup to produce compact salient objects.

V. EXPERIMENTS

The experiments compare SSA with state-of-the-art image- and video-based models on VOS to assess both the dataset and the baseline.

  • The benchmark includes models categorized as image-based or video-based, and classic unsupervised or non-deep learning, deep learning, or unsupervised methods.

A. Settings

The evaluation tests SSA and 31 state-of-the-art models on VOS using video-balanced metrics derived from per-video averages.

  • A. Settings: Thirty-two models, including SSA, are evaluated on VOS using Recall, Precision, Fβ, and MAE.The benchmark covers image-based classic, image-based deep, and video-based unsupervised categories.
  • A. Settings: MAR, MAP, and MAE are computed per video before averaging, preventing long videos from dominating the evaluation.Fβ combines MAR and MAP, with β^2 set to 0.3.
  • A. Settings: SSA produces valid results on 187 videos because the NLC executable fails on some inputs.The valid outputs cover 91 VOS-E videos and 96 VOS-N videos.
  • A. Settings: Adaptive thresholds convert grayscale saliency maps into binary masks for computing MAP, MAR, and Fβ.The threshold is twice the average saliency, capped at the map's maximum when necessary.

B. Model Benchmarking

SSA outperforms the compared models on key benchmark measures, while the results expose precision–recall trade-offs and challenges from temporal modeling and training-data dependence.

  • B. Model Benchmarking: SSA outperforms 30 state-of-the-art models in Fβ and outperforms all compared deep models in MAP.SSA uses no ground-truth data, whereas the deep baselines are trained or fine-tuned with large annotated image datasets.
  • B. Model Benchmarking: SSA achieves the best precision on VOS and both subsets, with MAP = 0.764 on VOS, despite lower MAR than some unsupervised image-based models.The comparison reflects a precision–recall trade-off in which SSA suppresses distractors more effectively.
  • B. Model Benchmarking: The benchmark highlights a challenge for VOS: improving recall while maintaining SSA's high precision.The paper notes that Fβ emphasizes precision more than recall, so higher recall alone may not improve the overall score.
  • B. Model Benchmarking: Some video-based models perform worse than image-based models because temporal information can act as noise under global camera motion and static objects.Manual parameter tuning on small datasets may also overfit video scenarios and propagate incorrect results.
  • B. Model Benchmarking: Image-based deep models generally outperform classic or unsupervised image-based models, with performance tending to improve with more training data and sources except for DCL.The paper relates this pattern to deep models' ability to exploit massive training data.

C. Performance Analysis of SSA

Additional experiments examine SSA's scalability, component contributions, temporal context, speed, and failure modes, showing strong transfer but substantial difficulty on realistic videos.

  • C. Performance Analysis of SSA: The component, speed, and failure analyses use cue-ablation, runtime, and qualitative experiments to expose SSA's design trade-offs and remaining bottlenecks.The study includes pixel, superpixel, and object cues, component timing, and comparisons with other video-based models.
  • C. Performance Analysis of SSA: SSA ranks second on ViSal without fine-tuning, behind DHSNet, indicating competitive cross-dataset scalability.Its MAE ranking improves relative to VOS, while the unsupervised architecture is described as generalizable to new scenarios.
  • C. Performance Analysis of SSA: Using one subsequent frame raises SSA's Fβ from 0.735 at W=0 to 0.755 at W=1.The experiment compares temporal windows W = 0, 1, 2, 4, 8, and 15.
  • C. Performance Analysis of SSA: SSA reaches Fβ = 0.850 on VOS-E but only 0.665 on VOS-N, with the decline mainly attributed to low recall.Real-world videos with more complex motion and appearance conditions are substantially harder than simple videos.
  • C. Performance Analysis of SSA: SSA's failure cases show that local spatiotemporal correspondences are insufficient when salient-object identity requires information from the entire video.The paper identifies global-perspective saliency computation as a needed direction for challenging sequences.

D. Discussion

The discussion presents video-based SOD as substantially harder than image-based SOD and positions VOS as a benchmark for studying spatiotemporal generalization and real-world difficulty.

  • D. Discussion: Video-based SOD is much more challenging than image-based SOD because strong performance requires temporal information from local and global perspectives.The conclusion states that even state-of-the-art image-based models remain far from perfect without such information.
  • D. Discussion: VOS-E provides a baseline for extending image-based models into the spatiotemporal domain, while VOS-N targets realistic deployment scenarios.The conclusion identifies small, fast-moving objects, poor lighting, and cluttered dynamic backgrounds as challenges in VOS-N.
  • D. Discussion: VOS combines human fixations with manually annotated objects across videos to reduce ambiguity in defining salient objects spatiotemporally.Its 200 videos support benchmarking differences and correlations between image-based and video-based SOD.
  • D. Discussion: The paper presents VOS and unsupervised saliency-guided stacked autoencoders as resources for benchmarking future video-based SOD models.The benchmark results are intended to reveal challenges, differences, and correlations across image-based and video-based approaches.
Loading 1611.00135v2…