PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Rich and Efficient Spatial Temporal Interactions for Real Time Video Salient Object Detection

Chenglizhao Chen, Guotao Wang, Chong Peng, Dingwen Zhang, Yuming Fang, Hong Qin

arXiv:2008.02973v1cs.CVcs.LG

TL;DR

Video saliency methods often combine spatial and temporal branches, but optical-flow computation can be costly and prior approaches may limit interaction between the branches. This paper integrates a lightweight temporal model into multi-scale spatial decoding and enables reciprocal branch interaction. The method is reported to achieve real-time detection at 50 FPS.

  • Problem

    Video saliency methods combine spatial color cues and temporal motion cues, but optical-flow computation is costly and some approaches subordinate temporal cues to spatial features.

    Source: §I · p. 1

  • Method

    The network inserts a lightweight temporal model into decoder layers to use multi-scale spatial features while sensing temporal information, with both branches interacting.

    Source: §I · p. 2

  • Results

    50 FPS is reported for real-time video saliency detection with the lightweight temporal module.

    Source: §I · p. 2

  • Takeaways & Limitations

    The temporal branch can facilitate spatial feature computation recurrently, supporting the paper’s full-interaction design.

    Source: §I · p. 2

    With only three consecutive input frames, the method may miss regions that remain static for long periods.

    Source: D. Limitations · p. 9

Abstract

from arXiv · show

The current main stream methods formulate their video saliency mainly from two independent venues, i.e., the spatial and temporal branches. As a complementary component, the main task for the temporal branch is to intermittently focus the spatial branch on those regions with salient movements. In this way, even though the overall video saliency quality is heavily dependent on its spatial branch, however, the performance of the temporal branch still matter. Thus, the key factor to improve the overall video saliency is how to further boost the performance of these branches efficiently. In this paper, we propose a novel spatiotemporal network to achieve such improvement in a full interactive fashion. We integrate a lightweight temporal model into the spatial branch to coarsely locate those spatially salient regions which are correlated with trustworthy salient movements. Meanwhile, the spatial branch itself is able to recurrently refine the temporal model in a multi-scale manner. In this way, both the spatial and temporal branches are able to interact with each other, achieving the mutual performance improvement. Our method is easy to implement yet effective, achieving high quality video saliency detection in real-time speed with 50 FPS.

I. INTRODUCTION AND MOTIVATION

Video saliency methods often rely on optical flow or make the temporal branch subordinate to spatial features, leaving speed and robustness challenges. The proposed network integrates a lightweight temporal model into multi-scale spatial decoding for interaction between both branches.

I. INTRODUCTION AND MOTIVATION

  • Prior methods commonly fuse spatial color cues with temporal motion cues, while recent approaches often subordinate temporal cues to spatial features.Motion information may be absent when a salient object remains static for a long period.Source: §I · p. 1
  • Optical-flow-based multi-scale fusion is limited by flow computation cost and occasionally inaccurate motion information.A ConvLSTM-based alternative is described as dependent on its preceding spatial branch, limited to about 10 FPS, and unable to use multi-scale spatial features fully.Source: §I · pp. 1–2
  • The network inserts a lightweight temporal model into each UNet decoder layer, combining multi-scale spatial features with temporal sensing.The temporal model uses sequential 3D convolutions and temporal shuffle operations; its features also facilitate spatial feature computation recurrently.Source: §I · p. 2
  • 50 FPS is reported for real-time detection using the temporal module instead of time-consuming optical flow.The authors describe optical flow’s computational cost as a major performance bottleneck for extremely fast video saliency detection.Source: §I · pp. 1–2

II. RELATED WORK

Prior work developed hand-crafted and deep-learning approaches to video saliency, while the proposed network places lightweight temporal models at each decoder layer to integrate multi-scale spatial and temporal features.

A. Handcraft Feature Based Methods

  • Hand-crafted methods estimate temporal information through spatiotemporal contrast, similarity propagation, geodesic measurements, gradient flow, and long-term consistency.Examples include superpixel-wise contrast histograms and intra-frame similarity matrices.Source: A. Handcraft Feature Based Methods · p. 2

III. SPATIOTEMPORAL NETWORK OVERVIEW

  • The proposed network uses a UNet [15] baseline and assigns a temporal model to each decoder layer to fuse multi-scale spatial features with temporal information.The temporal model also receives spatial attention maps, and its spatiotemporal features are recurrently integrated into the next decoder layer.Source: Figure 2 · p. 3, §III · pp. 3–4

B. Deep Learning Based Methods

  • Deep-learning methods use two-stream networks, 3D or 2D convolutions, ConvLSTMs, fixation cues, or optical flow, but prior approaches leave limitations in spatial-temporal interaction.The cited multi-task method uses its temporal branch to improve its spatial branch, but not the reverse.Source: B. Deep Learning Based Methods · pp. 2–3

A. Fast 3D Convolution

The temporal model applies sequential 3D convolutions to spatial features, using cyclic padding and feature additions to capture temporal information while addressing boundary-detail degradation and padding cost.

A. Fast 3D Convolution

  • Cyclic padding lets sequential 3D convolutions reuse other frames as padding, avoiding conventional-padding degradation and adding temporal information without extra computation.The method also adds 2D-convolution features to spatiotemporal features to address loss of fine spatial detail and blurred object boundaries.Source: A. Fast 3D Convolution · p. 4
  • The temporal model applies sliding-window 3D convolutions to three-frame spatial features, using three sequential convolutions because a single convolution was found insufficient for accurate temporal information.The input features are denoted T_i,j, and the model outputs spatiotemporal features ST_i,j.Source: A. Fast 3D Convolution · p. 4
  • Repeating the three input frames on the GPU and applying a sliding window implements cyclic padding five times faster than conventional feature reorganization.The repeat-and-window scheme supports cyclic padding without reorganizing the input data.Source: A. Fast 3D Convolution · p. 4

B. Fast Temporal Shuffle

The temporal shuffle addresses spatial bias in the temporal model by swapping deep features between consecutive frames. This lightweight operation mixes temporally neighboring spatial information into the current frame to strengthen temporal sensing.

B. Fast Temporal Shuffle

  • It counters the spatial bias of 3D kernels, whose temporal information comes from the kernels’ third dimension.The adopted kernel is described as 3×3 spatial by 3 temporal.Source: B. Fast Temporal Shuffle · pp. 4–5
  • Figure 4 maps the 64-channel input features of three consecutive frames to output features after they pass through one another.Top Frame i denotes input features and Bottom Frame i denotes output features.Source: Figure 4 · p. 5
  • The temporal shuffle swaps deep features between consecutive frames to enhance the temporal information sensed by the model.The paper describes swapping features from frame 1 with frame 2 and likewise swapping another feature pair.Source: B. Fast Temporal Shuffle · p. 5
  • The shuffle reshapes 192 features into 64 groups of three, transposes the 64 × 3 arrangement to 3 × 64, then flattens it.This inserts temporally neighboring spatial features into the current frame; the operation is repeated after the first and second 3D convolutions.Source: B. Fast Temporal Shuffle · p. 5

A. Datasets and Evaluation Criteria

The evaluation uses six public video-saliency datasets and three standard metrics. The model is pretrained and fine-tuned on image and video data, then trained jointly as a spatiotemporal network.

1) Evaluation Datasets.:

  • The evaluation covers DAVIS-T, SegTrack-V2, Visal, FBMS-T, VOS-T, and DAVSOD-T, using F-measure, S-measure, and MAE.The authors describe these as widely used public benchmarks and the metrics as commonly adopted.Source: 1) Evaluation Datasets · p. 5, 3) Training Set · p. 5

3) Training Set.:

  • The model is pretrained on 9.5K images and fine-tuned on 7.5K video samples, without DAVSOD fixation data.The image set combines DUTOMRON, HKU-IS, and MSRA10K; video data includes DAVIS-TR and DAVSOD.Source: 3) Training Set · p. 5

4) Training Details.:

  • Training first pretrains the spatial branch on 17.5K image and video samples, then trains the full spatiotemporal model for about 8,500 epochs.Each static image is copied three times for the model’s three-frame input; batch size decreases from 16 to 4 for spatiotemporal training.Source: 4) Training Details · p. 5
  • Training uses SGD with momentum 0.9, weight decay 5e-4, and initial learning rate 5e-3, with horizontal-flip and video-frame-rate augmentation.Video frame intervals are resampled from {0,1,2,3,4,5,6}.Source: 4) Training Details · p. 5

B. Performance Comparisons

The method is compared with 23 saliency detectors across video and image methods. It ranks among the top three on tested datasets, with its strongest reported results on SegTrack-V2 and VOS-T.

B. Performance Comparisons

  • The comparison includes 23 methods: 20 video-saliency detectors and three image-saliency detectors.The listed image methods are CPD, PoolNet, and EGNet.Source: B. Performance Comparisons · p. 5

1) Quantitative Evaluation:

2) Discussion:

  • The authors suggest that adding FBMS-T and DAVSOD fixation data to training might yield more competitive results on FBMS and DAVSOD.They state that full-leading performance on every tested dataset is not their central focus.Source: 1) Quantitative Evaluation · p. 6

1) Quantitative Evaluation:

  • Qualitative comparisons show an advantage in cluttered scenes with moderate motion, including the breakdance sequence.Figure 5 compares the proposed method with video- and image-saliency methods alongside the frame and ground truth.Source: 1) Quantitative Evaluation · p. 6

2) Discussion:

  • The reported wins span SegTrack-V2, dominated by temporal information, and VOS-T, dominated by spatial information.The authors describe these results as showing the method’s robustness.Source: 2) Discussion · p. 6

3) Efficiency Comparison.:

The method achieves the highest reported FPS and has the lightest network among the compared models. Component evaluations report gains from temporal and multi-scale components, while identifying design choices for the 3D-convolution sequence and padding.

3) Efficiency Comparison.:

C. Component Evaluation

5) The Reasons Why We Choose A Small Feature Size For Our Spatial-branch:

The method uses multi-scale spatial information to retain sharp object boundaries while allowing a small spatial feature size. This contrasts with prior single-scale interactions, whose detections can have blurred boundaries.

5) The Reasons Why We Choose A Small Feature Size For Our Spatial-branch:

  • SSAV and PDBM use single-scale spatial-to-temporal interaction, and their detections are frequently associated with blurred object boundaries.They use attention to compensate for lost boundaries; their dilated convolutions generally require a 60*60 size.Source: 5) The Reasons Why We Choose A Small… · p. 9
  • Multi-scale side-outputs provide spatial information to the temporal branch and are intended to preserve sharp object boundaries.The method's attention module supplies location information, so the spatial branch can use a small feature size.Source: 5) The Reasons Why We Choose A Small… · p. 9

D. Limitations

The method senses temporal information from three consecutive frames at a time, which can leave long-static regions undetected. The conclusion describes a lightweight temporal model, 3D convolutions, and a fast temporal shuffle scheme, and reports better detection performance and speed than current SOTA methods.

D. Limitations

  • Three-frame input limits temporal sensing and can leave regions in long-period static states undetected.The authors identify long-term spatiotemporal information as a future direction for addressing this limitation.Source: D. Limitations · p. 9

VI. CONCLUSION

  • The method inserts a lightweight temporal model into each decoder layer to obtain multi-scale spatiotemporal deep features.The conclusion also describes sequential 3D convolutions and a fast temporal shuffle scheme for temporal sensing.Source: §VI · p. 9
  • The authors report that their method outperforms current SOTA methods in both detection performance and speed.Source: §VI · p. 9

Related papers

Loading 2008.02973v1…