Source-linked AI summary
Revisiting Video Saliency: A Large-scale Benchmark and a New Model
Wenguan Wang, Jianbing Shen, Fang Guo, Ming-Ming Cheng, Ali Borji
TL;DR
Video saliency lacks broad, standard benchmarks for predicting human fixations during dynamic free viewing. This paper introduces the DHF1K benchmark and a CNN-LSTM model with supervised static attention; across large evaluations, the model outperforms competing methods.
Problem
Dynamic free-viewing fixation prediction has been less explored, and existing video datasets lack standard benchmarks, variety, generality, scalability, and preserved test sets.
Method
The paper combines a CNN-convolutional LSTM architecture with supervised attention learned from static fixation datasets to encode spatial saliency for temporal prediction.
Results
The proposed model outperforms competing video saliency methods on DHF1K, Hollywood-2, and UCF sports.
Takeaways & Limitations
DHF1K supplies a diverse benchmark, while supervised static attention supports efficient dynamic saliency representation learning.
Takeaways & Limitations
Earlier dynamic fixation datasets are limited in variety, generality, scalability, or stimulus purpose, and often lack preserved test sets.
Abstract
from arXiv · showhide
In this work, we contribute to video saliency research in two ways. First, we introduce a new benchmark for predicting human eye movements during dynamic scene free-viewing, which is long-time urged in this field. Our dataset, named DHF1K (Dynamic Human Fixation), consists of 1K high-quality, elaborately selected video sequences spanning a large range of scenes, motions, object types and background complexity. Existing video saliency datasets lack variety and generality of common dynamic scenes and fall short in covering challenging situations in unconstrained environments. In contrast, DHF1K makes a significant leap in terms of scalability, diversity and difficulty, and is expected to boost video saliency modeling. Second, we propose a novel video saliency model that augments the CNN-LSTM network architecture with an attention mechanism to enable fast, end-to-end saliency learning. The attention mechanism explicitly encodes static saliency information, thus allowing LSTM to focus on learning more flexible temporal saliency representation across successive frames. Such a design fully leverages existing large-scale static fixation datasets, avoids overfitting, and significantly improves training efficiency and testing performance. We thoroughly examine the performance of our model, with respect to state-of-the-art saliency models, on three large-scale datasets (i.e., DHF1K, Hollywood2, UCF sports). Experimental results over more than 1.2K testing videos containing 400K frames demonstrate that our model outperforms other competitors.
1. Introduction
The paper addresses the lack of standard, diverse benchmarks for dynamic free-viewing fixation prediction by introducing DHF1K and an attentive CNN-LSTM model. The model uses supervised static attention to improve temporal saliency learning and outperforms previous methods.
- Motivation: Existing video saliency research lacks standard benchmarks with the breadth and evaluation controls available in static saliency.The paper identifies missing widespread benchmarks and emphasizes the need for preserved test data to reduce overfitting.
- Benchmark: DHF1K provides a standard benchmark of 1K diverse videos with per-frame fixation annotations from 17 observers.The dataset covers varied scenes, motion patterns, object categories, activities, and attributes, with a preserved test set for evaluation.
- Model: The proposed model combines CNN frame features, convolutional LSTM sequence modeling, and supervised attention learned from large-scale image saliency datasets.Attention enhances spatially informative CNN features and helps disentangle spatial and temporal factors of dynamic attention.
- Model: The attentive CNN-LSTM architecture explicitly incorporates static attention while learning dynamic saliency representations across successive frames.The design leverages both static and dynamic fixation data and is intended to improve training efficiency and generalization.
- Results: The model significantly outperforms previous methods in evaluations spanning Hollywood-2, UCF sports, and DHF1K.The paper presents the evaluation as a comprehensive comparison across existing and newly introduced video saliency datasets.
2. Related Work
Prior dynamic eye-tracking datasets are constrained in variety, generality, scale, or viewing purpose, while earlier computational approaches rely heavily on handcrafted features. The paper motivates supervised attention as a way to incorporate static saliency information into dynamic fixation prediction.
- Video Eye-Tracking Datasets: Existing dynamic fixation datasets often lack variety, generality, and scalability, limiting their coverage of video stimuli.The paper identifies these constraints as common across several previously released datasets.
- Video Eye-Tracking Datasets: Hollywood-2 contains many videos but is restricted to human actions and movie scenes, with viewing biased toward task-driven observation.In a sample of 1,000 frames, 84.5% of fixations were located around faces.
- Video Eye-Tracking Datasets: UCF sports comprises sports-action videos and similarly uses task-aware viewing instructions rather than unconstrained free viewing.The paper reports that 82.3% of fixations in 1,000 sampled frames fell inside human body areas.
- Video Eye-Tracking Datasets: DIEM provides free-viewing fixation data from around 50 observers per video but remains limited in coverage and scale.Its 84 videos come from publicly accessible advertisements, documentaries, sport events, and movie trailers.
- Computational Models for Fixation Prediction: Earlier dynamic fixation models combine static features with temporal signals but are largely constrained by feature engineering and handcrafted representations.Only a few deep learning-based video saliency models existed at the time described by the paper.
- Attention Mechanisms in Neural Networks: The proposed supervised attention module enhances intra-frame salient features so LSTM can model dynamic representations more easily.Unlike implicitly learned attention, it encodes static saliency information and can be trained using existing static saliency datasets.
3. DHF1K Dataset
DHF1K is a large-scale dynamic video eye-tracking dataset designed to support free-viewing fixation prediction across diverse content. It provides fixation annotations, dataset statistics, controlled capture conditions, and a held-out benchmark split.
- DHF1K contains 1K free-viewing videos with eye-tracking annotations from 17 observers.
- DHF1K provides statistics on motion patterns, main-object counts, scene illumination, and the number of people.
- The dataset was designed around diverse video content, including varied scenes, motion patterns, object categories, and activities.
- The eye-tracking experiment used binocular monitoring at 250 Hz on a 19-inch display, with a headrest maintaining an approximately 68 cm viewing distance.
- Participants viewed randomly ordered videos across ten non-overlapping sessions with five-second black-screen intervals between presentations.
- The benchmark split comprises 600 training videos, 100 validation videos with public fixation records, and 300 test videos with held-out annotations.
4. Our Approach
The approach extends CNN-LSTM video saliency modeling with supervised static attention, allowing convolutional features and convLSTM to represent spatial and temporal saliency. It also trains with static and dynamic fixation data using a multi-metric loss.
- CNN-LSTM architecture: The model combines CNN feature extraction, convLSTM temporal modeling, and an attention mechanism for video saliency prediction.The convLSTM preserves spatial information while modeling sequential dynamics.
- Attention module: The attention branch generates a 7×7 map, upsamples it fourfold to 28×28, and enhances 28×28×512 conv5-3 features.The attention map has an enlarged 260×260 receptive field before upsampling.
- Attention module: A residual attention connection combines original and enhanced CNN features before feeding them to the LSTM.This modification addresses the possibility that static attention alone may discard useful dynamic-saliency information.
- Attention module: The attention module learns explicit intra-frame static saliency from large-scale static fixation data, allowing convLSTM to focus on dynamic representations.This supervised design is intended to disentangle spatial and temporal factors of dynamic attention.
- Training objective: The loss combines KL divergence, correlation coefficient, and normalized scanpath saliency because no single metric fully captures saliency-map quality.The balance parameters are empirically set to α1 = α2 = 0.1.
- Training protocol: Training alternates video batches for dynamic prediction with image batches that supervise only the attention module using static fixation data.Video batches use 20 consecutive frames, while image batches are randomly sampled from existing static fixation datasets.
5. Experiments
The experiments evaluate the model across three video-saliency datasets and compare its components, training protocols, and performance with existing methods. Results consistently favor the proposed architecture, while showing that static data, temporal modeling, and global receptive fields contribute to performance.
- Performance Comparison: The proposed model consistently and significantly outperforms competing methods across all metrics on the 300-video DHF1K test set.The comparison includes quantitative results summarized in Table 5.
- Performance Comparison: On Hollywood-2, the model again performs significantly higher than other methods across various metrics, and performance increases with more training data.The experiments use 884 testing video sequences for this dataset.
- Performance Comparison: On UCF sports, the model provides consistently good results despite the dataset’s small and highly related videos, where 82.3% of fixations lie on human body areas.The authors relate limited training-data diversity to the dataset’s characteristics.
- Analysis: Deep learning dynamic models outperform classic dynamic methods, while classic dynamic methods do not outperform static counterparts and prior deep dynamic models remain limited.The authors attribute the latter limitation to complex dynamic attention and insufficient use of static saliency data.
- Ablation Analysis: Removing attention reduces AUC-J from 0.890 to 0.847, reducing static training stimuli from 10K to 5K reduces it to 0.877, and removing convLSTM also lowers performance.An enlarged receptive field improves performance by supporting prediction from a global view.
6. Discussion and Conclusion
The paper presents DHF1K as a large-scale benchmark for dynamic fixation prediction and proposes an attentive deep-learning model for video saliency. Across DHF1K, Hollywood-2, and UCF sports, the model outperforms competitors and is reported to be efficient.
- Benchmark: DHF1K contains 1K videos with representative instances, diverse contents, various motions, and human eye-tracking annotations.The benchmark is designed to facilitate research in video saliency modeling.
- Model: The proposed model uses supervised attention to capture static saliency information and help LSTM learn dynamic saliency representations over successive frames.The paper evaluates the model on DHF1K, Hollywood-2, and UCF sports.
- Conclusion: Experiments show that the proposed model outperforms competing methods and is quite efficient.The reported evaluation compares the model with previous attention models in dynamic scenes.