Source-linked AI summary
Motion Guided Attention for Video Salient Object Detection
Haofeng Li, Guanqi Chen, Guanbin Li, Yizhou Yu
TL;DR
Video salient-object detection must account for object motion, but existing methods do not explicitly model motion saliency or spatial context in optical flow. The paper addresses this with two end-to-end trained branches linked by motion-guided attention, and reports significant gains over state-of-the-art methods across broad benchmarks.
Problem
Video salient-object detection needs effective object-motion modeling, while existing methods may omit explicit motion cues or spatial context in optical-flow images.
Method
A multi-task network combines an appearance branch for still-image saliency with a motion branch for optical-flow saliency, using motion-guided attention to enhance appearance features.
Results
The method significantly surpasses existing state-of-the-art algorithms across a wide range of datasets and metrics.
Takeaways & Limitations
The proposed network establishes a strong video salient-object detection baseline using short-range context from the previous frame instead of long-range historical features.
Takeaways & Limitations
When only part of a salient object moves, still salient regions may be predicted as non-salient by the motion branch and suppressed in attention.
Abstract
from arXiv · showhide
Video salient object detection aims at discovering the most visually distinctive objects in a video. How to effectively take object motion into consideration during video salient object detection is a critical issue. Existing state-of-the-art methods either do not explicitly model and harvest motion cues or ignore spatial contexts within optical flow images. In this paper, we develop a multi-task motion guided video salient object detection network, which learns to accomplish two sub-tasks using two sub-networks, one sub-network for salient object detection in still images and the other for motion saliency detection in optical flow images. We further introduce a series of novel motion guided attention modules, which utilize the motion saliency sub-network to attend and enhance the sub-network for still images. These two sub-networks learn to adapt to each other by end-to-end training. Experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art algorithms on a wide range of benchmarks. We hope our simple and effective approach will serve as a solid baseline and help ease future research in video salient object detection. Code and models will be made available.
1. Introduction
The paper targets video salient-object segmentation by explicitly modeling object motion alongside appearance. It proposes two cooperating branches connected by motion-guided attention and reports broad benchmark gains.
- Motivation: Object motion affects visual attention and provides spatial-coherence cues beyond appearance in video salient-object detection.The paper links motion to frame-to-frame visual contrast, grouping of similarly displaced regions, and easier salient-object segmentation.
- Motivation: Existing methods insufficiently exploit motion patterns, while graph-based approaches rely on handcrafted features and limited training-data mining.The paper states that these limitations hinder adaptive modeling of motion contrast and high-level object semantics in complicated scenes.
- Proposed approach: The method uses short-range context from the previous frame rather than long-range historical features required by ConvLSTM-based algorithms.The authors present this design as a strong baseline for video salient-object detection.
- Proposed approach: Motion-guided attention modules attend to and enhance appearance features using motion features or motion saliency.The modules combine residual learning with spatial and channel-wise attention.
- Proposed approach: The proposed architecture uses an appearance branch, a motion branch, and attention modules that bridge them.The appearance branch performs salient-object detection on still images, while the motion branch detects motion saliency from optical-flow images.
- Results: The proposed network significantly surpasses existing state-of-the-art algorithms across a wide range of datasets and metrics.The experiments evaluate both the attention modules and the overall network.
2. Related Work
Related work spans deep video-saliency models, attention mechanisms, and optical-flow estimation. The paper positions its contribution as using attention to exploit motion’s complementary enhancement of appearance contrast.
- Video salient object detection: Deep video salient-object detection methods include region-wise labeling models and dense pixel-wise labeling models.The dense-labeling category includes fully convolutional and recurrent-neural-network approaches.
- Visual attention: Computer-vision attention mechanisms highlight positions or nodes according to their importance and include spatial, recurrent, and residual-attention designs.The cited examples span image captioning, image classification, and discriminative-region exploration.
- Paper positioning: The paper explores the complementary enhancement of motion information on appearance-contrast modeling through multiple attention schemes.It presents this as a first exploration of that effect in the cited line of work.
- Optical flow: Optical flow represents pixel-level motion between consecutive video frames and supports motion-based modeling.The related work reviews methods for estimating small and large displacements from consecutive frames.
3. Method
The method uses motion information to guide appearance features through several attention modules, while a two-branch architecture separates appearance and motion processing. The branches are connected by motion-guided modules during video SOD, with optical flow providing motion input.
- Motion-guided attention: Motion saliency maps attend appearance features through element-wise multiplication, but naive multiplication can suppress still salient regions.Regions moving like the background may receive zero motion saliency, suppressing corresponding appearance features and fragmenting salient objects.
- Motion-guided attention: MGA-m adds the original appearance feature to multiplication-based attention, highlighting salient motion without discarding still salient areas.The residual addition complements features incorrectly suppressed by motion-map multiplication.
- Motion-guided attention: MGA-t transforms motion features with a 1 × 1 convolution before applying element-wise multiplication-and-addition attention to appearance features.The convolution aligns the motion tensor shape with the appearance tensor.
- Motion-guided attention: MGA-tm converts motion features into H × W spatial attention maps, whereas MGA-t jointly represents spatial and channel-wise attention in a 3D tensor.The motion feature is transformed with a 1 × 1 convolution producing one output channel before Sigmoid activation.
- Motion-guided attention: MGA-tmc first emphasizes spatial locations with motion, then selects channels using global average pooling and normalized channel weights, and finally adds the input feature.Its channel weights target attributes such as edges, boundaries, colors, texture, and semantics.
- Network architecture: The network combines an appearance branch, a lighter motion branch, pretrained optical-flow estimation, and attention modules connecting the branches.The appearance and motion branches use encoder, ASPP, and decoder components; MGA-{0-4} connect encoder-side features while MGA-5 is also used in the architecture.
4. Experiments
Experiments evaluate the proposed network, its motion-guided attention modules, and its multi-task training scheme across video benchmarks and ablations. The method achieves strong benchmark performance, while architectural and training comparisons support the proposed design choices.
- Comparison with the state-of-the-art: The method achieves the lowest MAE and highest S-m and maxF on DAVIS, FBMS, and ViSal.On DAVIS, it exceeds PDB by 3.4% S-m and 4.0% maxF; on FBMS, it exceeds PDB by 6.1% S-m and 8.1% maxF.
- Comparison with the state-of-the-art: ‘Ours+CRF’ achieves the best J Mean and F Mean on DAVIS and the best mIoU on FBMS among compared unsupervised video segmentation methods.
- Effectiveness of the proposed network architecture: 6.6% S-m separates dual branch+MGA-D from the appearance branch on FBMS, while the dual-branch model also exceeds the motion branch by 14.4% S-m.Using attention at both encoder and decoder sides further improves FBMS maxF by 1.9% over MGA-D and 1.7% over MGA-E.
- Effectiveness of the proposed motion guided attention: 2.4% S-m and 3.8% maxF are the DAVIS gains of the motion-guided attention modules over the best naive fusion, Add.The comparison evaluates concatenation, multiplication, and addition as alternative fusion operators.
- Effectiveness of the proposed motion guided attention: The encoder MGA-tmc and decoder MGA-m are the strongest tested module choices, with MGA-m exceeding D-Add by 0.9% S-m and 0.8% maxF on FBMS.E-MGA-tm exceeds E-Add by 1.7% maxF on DAVIS, while E-MGA-t exceeds E-Add by 1.6% S-m on FBMS.
- Effectiveness of the proposed training scheme: The multi-task training scheme Tma exceeds the second-best Ta by 1.7% maxF on DAVIS and 1.9% maxF on FBMS.Tma pretrains the motion and appearance branches on their corresponding subtasks before end-to-end training.
5. Conclusions
The motion guided attention network establishes a state-of-the-art baseline for video salient object detection by modeling how salient motion patterns influence object saliency.
- The motion guided attention network models how salient motion patterns affect object saliency through attention.Its motion guided attention modules instantiate this mechanism within the network.