Source-linked AI summary
TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection
Kyle Min, Jason J. Corso
TL;DR
Video saliency models need to jointly use spatial and temporal information, which prior approaches do not do during final prediction. TASED-Net uses a 3D fully-convolutional encoder-decoder with temporal aggregation and Auxiliary pooling, significantly outperforming prior state-of-the-art methods on three major datasets. The authors conclude that limited past-frame conditioning and joint spatial-temporal processing are effective, while noting performance limitations for scattered or unstable fixation points and little benefit from deeper spatial decoding.
Problem
Existing video saliency models fail to jointly process spatial decoding and temporal aggregation when predicting saliency maps from extracted features.
Method
TASED-Net encodes an input clip spatiotemporally, decodes features spatially, aggregates temporal information, and uses Auxiliary pooling to obtain compatible max-unpooling switches.
Results
TASED-Net significantly outperforms previous state-of-the-art methods on DHF1K, Hollywood2, and UCFSports.
Takeaways & Limitations
Joint fully-convolutional spatial decoding and temporal aggregation, conditioned on a limited amount of past information, are effective for video saliency detection.
Takeaways & Limitations
Performance is worse than ACLNet for videos with widely scattered fixation points, and adding deeper spatial decoding slightly worsens performance.
Abstract
from arXiv · showhide
TASED-Net is a 3D fully-convolutional network architecture for video saliency detection. It consists of two building blocks: first, the encoder network extracts low-resolution spatiotemporal features from an input clip of several consecutive frames, and then the following prediction network decodes the encoded features spatially while aggregating all the temporal information. As a result, a single prediction map is produced from an input clip of multiple frames. Frame-wise saliency maps can be predicted by applying TASED-Net in a sliding-window fashion to a video. The proposed approach assumes that the saliency map of any frame can be predicted by considering a limited number of past frames. The results of our extensive experiments on video saliency detection validate this assumption and demonstrate that our fully-convolutional model with temporal aggregation method is effective. TASED-Net significantly outperforms previous state-of-the-art approaches on all three major large-scale datasets of video saliency detection: DHF1K, Hollywood2, and UCFSports. After analyzing the results qualitatively, we observe that our model is especially better at attending to salient moving objects.
1. Introduction
Video saliency detection models human gaze in dynamic scenes, but prior LSTM-based approaches do not jointly process spatial decoding and temporal aggregation. TASED-Net addresses this gap with a 3D fully-convolutional encoder-decoder and demonstrates strong performance across three large-scale datasets.
- Video saliency detection models human gaze fixation patterns in dynamic scenes and supports applications including surveillance, captioning, and compression.
- Existing LSTM-based models separate spatial decoding from temporal aggregation or use only one process, preventing joint use of spatiotemporal information.
- TASED-Net is an end-to-end 3D fully-convolutional encoder-prediction network that jointly decodes spatial features and aggregates temporal information.
- The encoder extracts low-resolution spatiotemporal features, while the prediction network spatially upsamples them and temporally aggregates information into a full-resolution saliency map.
- Auxiliary pooling adds extra max-poolings to generate properly sized switches for max-unpooling layers whose temporal dimensions are otherwise incompatible.
- TASED-Net significantly outperforms previous state-of-the-art baselines on DHF1K, Hollywood2, and UCFSports.
2. Related Work
Prior video saliency models use optical flow, recurrent temporal aggregation, or spatiotemporal 3D features, while 2D encoder-decoder advances motivate spatial reconstruction. TASED-Net builds on these lines with a 3D encoder-decoder design and Auxiliary pooling.
- Previous video saliency models rely on optical flow or LSTM to incorporate temporal information, using separate temporal streams or recurrent aggregation.
- STSConvNet uses a two-stream architecture in which optical-flow temporal information is processed independently from spatial information.
- RMDN extracts spatiotemporal features with C3D and aggregates temporal information over the long term using a subsequent LSTM.
- 2D encoder-decoder networks established spatial feature reconstruction approaches, including learnable upsampling through transposed convolution.
- TASED-Net combines a 3D convolutional encoder with spatial decoding and introduces Auxiliary pooling, inspired by encoder-decoder reconstruction methods.
- 3D ConvNets extend successful 2D architectures and have achieved state-of-the-art results in action recognition, motivating their use for spatiotemporal feature extraction.
3. Approach
TASED-Net encodes a fixed-length video clip into low-resolution spatiotemporal features, then jointly performs spatial decoding and temporal aggregation to produce one full-resolution saliency map. Its architecture uses sliding-window prediction, Auxiliary pooling for spatial unpooling, and a late two-step temporal aggregation strategy.
- 3.1. Architecture Overview: TASED-Net assumes each frame’s saliency can be predicted from a fixed number T of consecutive past frames and predicts the last frame of each input clip.Frame-wise maps are generated with a sliding window; the first T−1 frames are handled by reversing the chronological order of initial clips, provided N >= 2T −1.
- 3.1. Architecture Overview: The encoder extracts deep low-resolution spatiotemporal features, while the prediction network spatially decodes them and aggregates temporal information into a full-resolution map.The prediction network jointly processes spatial and temporal information without upsampling the temporal dimension.
- 3.3. Auxiliary pooling: In the illustrated 2 × 2 × 2 example, P1 uses 2 × 1 × 1 max-pooling, P2 uses 1 × 2 × 2 max-pooling, and Us unpools spatially with a 1 × 2 × 2 kernel.P1 produces the temporally reduced map p, while P2 stores switches s used by the corresponding decoder unpooling layer.
- 3.3. Auxiliary pooling: Auxiliary pooling adds two sequential pooling operations that reduce temporal length and store spatial switches compatible with decoder max-unpooling.These poolings store switches outside the main data stream, allowing spatial reconstruction despite incompatible encoder and decoder temporal dimensions.
- 3.4. Temporal aggregation strategy: Temporal aggregation reduces the time dimension to 1 while upscaling quarter-resolution features to full spatial resolution through temporal and transposed convolutions.The compared strategies order these operations differently; late two-step aggregation applies temporal convolutions after each spatial upsampling and performs best.
4. Evaluation
TASED-Net is evaluated on three video-saliency datasets using standard metrics, model variants, and comparisons with prior methods. It performs strongly overall, while qualitative and ablation analyses identify where its design helps and where evaluation or model capacity remains limited.
- Experimental setup: The evaluation uses DHF1K, Hollywood2, and UCFSports, with NSS, CC, SIM, AUC-J, and s-AUC as reported metrics.DHF1K is the main benchmark because it contains diverse scenes, objects, motion, and backgrounds.
- DHF1K evaluation: T = 32 with late two-step temporal aggregation performs best across most DHF1K validation metrics.This configuration uses 21.2M parameters and 63.2G FLOPs.
- DHF1K evaluation: Removing Kinetics pre-training reduces performance by 0.5 NSS, 0.06 CC, 0.04 SIM, and 0.015 AUC in most cases.The reported model therefore uses Kinetics pre-training together with T = 32 and late two-step aggregation.
- Benchmark results: TASED-Net significantly outperforms previous state-of-the-art methods across all DHF1K evaluation metrics and achieves high scores across most metrics on Hollywood2 and UCFSports.The DHF1K test comparison also reports a smaller model size than ACLNet: 82MB versus 252MB.
- Qualitative analysis: On 100 DHF1K validation videos, TASED-Net beats ACLNet by more than 0.5 NSS on 37 videos, whereas ACLNet does so on 7.Qualitatively, TASED-Net attends more to salient moving objects and less to distracting background objects, producing less blurry maps in many cases.
- Limitations and observations: Performance can worsen when ground-truth fixations are widely scattered, and the authors call for more subjects and more diverse scenes in such cases.The authors also observe that deeper spatial decoding slightly worsens performance, possibly because spatial decoding is less important or because of overfitting.
- Ablation analysis: Auxiliary pooling is necessary: replacing max-unpooling with trilinear upsampling or transposed convolutions performs poorly.The operation supplies properly sized switches needed by the max-unpooling layers.
5. Conclusion
TASED-Net combines spatial decoding with temporal aggregation in a fully convolutional architecture for video saliency detection. The conclusion reports strong benchmark performance and validates auxiliary pooling through variant analysis.
- TASED-Net spatially decodes encoder features while jointly aggregating temporal information to produce a single full-resolution prediction map.
- The architecture uses Auxiliary pooling to leverage max-unpooling layers for reconstruction.
- TASED-Net significantly outperforms previous state-of-the-art methods on major video saliency detection datasets.
- Variant analyses show that Auxiliary pooling is necessary and effective.