PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Image and Video Saliency Modeling

Richard Droste, Jianbo Jiao, J. Alison Noble

arXiv:2003.05477v3cs.CVcs.LG

TL;DR

Image and video saliency are usually modeled separately despite targeting the same human-attention process. UNISAL unifies them with domain-adaptive modules, achieving leading video performance, competitive image performance, faster runtime, and substantially smaller models.

  • Problem

    Image and video saliency are usually treated as separate tasks despite both modeling human visual attention, while some video models cannot accept image inputs.

  • Method

    UNISAL jointly trains a lightweight encoder-RNN-decoder network on image and video data using four domain-adaptation techniques to address inter-dataset shifts.

  • Results

    UNISAL achieves state-of-the-art performance on all video datasets, image performance on par with the state of the art, 5–20-fold smaller models, and faster runtime.

  • Takeaways & Limitations

    Accounting for domain shifts is crucial for unifying static and dynamic saliency modeling while retaining competitive performance and computational efficiency.

Abstract

from arXiv · show

Visual saliency modeling for images and videos is treated as two independent tasks in recent computer vision literature. While image saliency modeling is a well-studied problem and progress on benchmarks like SALICON and MIT300 is slowing, video saliency models have shown rapid gains on the recent DHF1K benchmark. Here, we take a step back and ask: Can image and video saliency modeling be approached via a unified model, with mutual benefit? We identify different sources of domain shift between image and video saliency data and between different video saliency datasets as a key challenge for effective joint modelling. To address this we propose four novel domain adaptation techniques - Domain-Adaptive Priors, Domain-Adaptive Fusion, Domain-Adaptive Smoothing and Bypass-RNN - in addition to an improved formulation of learned Gaussian priors. We integrate these techniques into a simple and lightweight encoder-RNN-decoder-style network, UNISAL, and train it jointly with image and video saliency data. We evaluate our method on the video saliency datasets DHF1K, Hollywood-2 and UCF-Sports, and the image saliency datasets SALICON and MIT300. With one set of parameters, UNISAL achieves state-of-the-art performance on all video saliency datasets and is on par with the state-of-the-art for image saliency datasets, despite faster runtime and a 5 to 20-fold smaller model size compared to all competing deep methods. We provide retrospective analyses and ablation studies which confirm the importance of the domain shift modeling. The code is available at https://github.com/rdroste/unisal

1 Introduction

UNISAL frames image and video saliency as a unified modeling task, identifying domain shift as the central obstacle to joint learning and addressing it with four domain-adaptation techniques. One shared model achieves state-of-the-art video performance and competitive image performance while reducing model size and runtime.

  • Domain adaptation: Domain shift between image and video data, and among video datasets, is identified as the main challenge for joint modeling.The authors propose domain adaptation to address these identified sources of shift.
  • Unified modeling: UNISAL is presented as the first unified framework for modeling image and video visual saliency.It is designed to model both data types coequally within a simple, lightweight neural network.
  • Domain adaptation: Four techniques—Domain-Adaptive Priors, Domain-Adaptive Fusion, Domain-Adaptive Smoothing, and Bypass-RNN—enable strong shared features.These techniques are integrated into UNISAL to model saliency across image and video domains.
  • Efficiency: 5 to 20-fold smaller model size and faster runtime distinguish UNISAL from existing deep saliency models.The paper reports these efficiency advantages while maintaining the stated video and image saliency performance.

2 Related Work · 3 Unified Image and Video Saliency Modeling

The paper reviews separate image- and video-saliency approaches, then introduces UNISAL, a lightweight unified encoder-RNN-decoder model that addresses domain shifts through adaptive priors, fusion, smoothing, normalization, and a Bypass-RNN.

  • 2 Related Work: Image saliency progressed from low-level bottom-up features to deep networks learning high-level top-down features.
  • Image Saliency Modeling · Video Saliency Modeling: Video saliency similarly evolved from low-level spatial-temporal statistics to deep ConvLSTM-based models and the DHF1K benchmark.
  • 3.1 Domain-Shift Modeling: The proposed analysis examines domain shifts between image and video data and among DHF1K, Hollywood-2, UCF Sports, and SALICON.
  • Domain-Adaptive Batch Normalization · Domain-Adaptive Fusion · Domain-Adaptive Smoothing: Dataset differences include inaccurate shared batch-normalization statistics, distinct center biases, task-driven versus free-viewing fixations, and heterogeneous saliency-map sharpness.
  • 3.2 UNISAL Network Architecture · Domain-Adaptive Fusion: UNISAL uses a MobileNet-V2 encoder, learned Gaussian priors, a Bypass-RNN, skip-connected decoder, and domain-adaptive fusion and smoothing layers.
  • Encoder Network · Gaussian Prior Maps: The model uses unconstrained Gaussian priors, initializes 16 maps across broad prior configurations, and injects them before the RNN and decoder.
  • Bypass-RNN: The Bypass-RNN adds recurrent outputs residually for video inputs while automatically bypassing recurrence for static image batches.
  • 3.3 Domain-Aware Optimization · Assimilated Frame Rate: Training accommodates dataset-specific aspect ratios and assimilates all videos to 6 fps by sampling every 5th DHF1K frame and every 4th frame from the other datasets.

4 Experiments

Experiments evaluate UNISAL jointly trained on image and video saliency data across six video training settings and standard image benchmarks. Results show strong cross-dataset generalization, substantial contributions from domain adaptation, and leading efficiency in model size and runtime.

  • Experimental setup: UNISAL is jointly trained using SALICON, Hollywood-2, UCF Sports, and DHF1K data, with six settings ranging from single-dataset to all-dataset training.The six settings separately assess dataset-specific training, cross-dataset video training, and joint image-video training.
  • Quantitative results: Adding SALICON to joint video training improves most DHF1K and UCF Sports metrics, while Hollywood-2 decreases by less than 1%.This result indicates that image saliency data generally benefits video saliency performance, with a small exception on Hollywood-2.
  • Quantitative results: UNISAL performs on par with state-of-the-art image saliency models on both SALICON and MIT300.The evaluation also compares dynamic video saliency models on the static SALICON benchmark.
  • Ablation analysis: Each proposed component provides a considerable performance increase, with domain-adaptive operations contributing the most on DHF1K and SALICON.The ablation evaluates Gaussian priors, recurrent and skip connections, smoothing, domain-adaptive operations, and domain-aware optimization.
  • Domain-shift analysis: Video-domain feature statistics correlate at 82%–83%, whereas SALICON-video correlations are below 3%, confirming dataset feature-distribution shifts.SALICON fusion weights also cluster near zero relative to video weights, while domain-adaptive Gaussian priors learn deviations from initialization.
  • Efficiency analysis: UNISAL is over 5× smaller than TASED-Net and processes video at 0.43 s per frame on CPU, or 2.3 fps.The lightweight design uses an MNet V2 encoder, separable convolutions, a low-resolution cGRU bottleneck, and bilinear upsampling.

5 Discussion and Conclusion

The paper unifies static and dynamic saliency modeling with a lightweight UNISAL architecture whose domain-adaptive modules address inter-dataset domain shifts. This approach achieves state-of-the-art performance on video saliency datasets.

  • Unified modeling: UNISAL unifies static and dynamic saliency modeling through a simple, lightweight architecture designed to model both modalities coequally.The architecture integrates novel domain-adaptive modules to bridge the domain gap caused by different sources of inter-dataset shift.
  • Domain adaptation: Accounting for different sources of inter-dataset domain shift was crucial for bridging the domain gap.The paper addresses this through corresponding novel domain-adaptive modules integrated into UNISAL.
  • Results: UNISAL achieved state-of-the-art performance on video saliency datasets.This result was obtained using the unified static-and-dynamic saliency modeling approach.

(Supplementary Material)

The supplementary material identifies the authors’ University of Oxford affiliation and records the paper as arXiv:2003.05477v3, dated 7 November 2020.

  • The authors are affiliated with the University of Oxford, and the paper is listed as arXiv:2003.05477v3 from 7 November 2020.

1 Introduction

The supplementary material provides additional quantitative and qualitative results to clarify the proposed model for unified image and video saliency analysis. It covers qualitative results, cross-domain predictions, center-bias analysis, ablations, generalization, probabilistic benchmarks, evaluation details, and code.

  • Supplementary Material: The supplement adds quantitative and qualitative results for understanding the proposed unified image and video saliency model.These results are presented as supplementary analyses of the model.
  • Qualitative Results: Additional qualitative results are provided for both video and image saliency.The supplement separately identifies qualitative video saliency and qualitative image saliency results.
  • Additional Analyses: The supplement also examines cross-domain predictions, center bias, ablations, cross-dataset generalization, probabilistic benchmark results, quantitative evaluation details, and code.These topics are listed across Sections 4–10 of the supplementary material.

2 Additional Qualitative Video Saliency Results

The section presents additional qualitative video saliency predictions, including comparisons with state-of-the-art methods across representative clips from three video saliency datasets.

  • 2 Additional Qualitative Video Saliency Results: Additional qualitative predictions compare UNISAL with state-of-the-art methods on representative clips from DHF1K, UCF Sports, and Hollywood-2.Results are provided in supplementary video 3601-supp.mp4, with frame-based examples shown in a figure.

3 Additional Qualitative Images Saliency Results

This section presents additional qualitative image-saliency predictions sampled from SALICON and MIT1003, with results shown in Figures 2 and 3.

  • Additional qualitative image-saliency prediction results are included beyond those presented in the main paper.
  • Representative images are sampled from the SALICON and MIT1003 datasets, with results shown in Figures 2 and 3, respectively.

4 Cross-Domain Predictions

This section analyzes how domain-adaptive modules affect saliency predictions on identical inputs across video and image datasets.

  • Cross-Domain Predictions: The analysis compares domain-adaptive module settings for visual saliency prediction on the same input.Video results are presented in the supplementary video, while image results are shown for SALICON and MIT1003.

5 Additional Center Bias Analysis · 6 Additional Ablation Studies · 7 SALICON Cross-Dataset Generalization

The additional analyses examine how UNISAL models dataset-specific center biases, how its domain-adaptive modules contribute individually, and how training-data choices affect SALICON generalization. Together, these sections extend evaluation across learned priors, ablations, and cross-dataset settings.

  • 5 Additional Center Bias Analysis: The center-bias analysis evaluates whether domain-adaptive learned Gaussian prior maps capture dataset-specific center biases.Averaged saliency targets approximate true center biases, while all-zero-input predictions reveal learned biases for each domain-adaptive module set.
  • 5 Additional Center Bias Analysis: Averaged saliency targets from each training dataset are used as approximations of the corresponding true center biases.The analysis compares these targets with saliency predictions generated from all-zero inputs.
  • 6 Additional Ablation Studies: The additional ablation study removes or adds individual domain-adaptive modules on DHF1K and SALICON using the main ablation metrics.The starting point is the baseline with modules through Smoothing, followed by individual additions.
  • 6 Additional Ablation Studies: The tested additions are DABN, DA-Gaussians, DA-Fusion, DA-Smoothing, and BypassRNN.The study is designed to analyze the respective effectiveness of these modules.
  • 6 Additional Ablation Studies: The ablation results show that each domain-adaptive module contributes differently.These comparisons are reported in Table 1 for DHF1K and SALICON.
  • 7 SALICON Cross-Dataset Generalization: The SALICON cross-dataset analysis evaluates UNISAL image-saliency prediction after training under six dataset configurations.Setting (vi), trained on DHF1K, Hollywood-2, UCF Sports, and SALICON, is the final model.
  • 7 SALICON Cross-Dataset Generalization: The six training settings range from training on each individual dataset to joint training on the three video datasets plus SALICON.The evaluation uses the standard SALICON benchmark pipeline and adds KLD as an additional metric.
  • 7 SALICON Cross-Dataset Generalization: Table 3 reports MIT300 results with probabilistic predictions using training setting (vi).This result is listed as part of the SALICON cross-dataset generalization subsection.

8 MIT300 Probabilistic Benchmark Results · 9 Details for Quantitative Evaluation

The paper reports MIT300 predictions using cross-validated fine-tuning and discusses metric-specific scoring, while documenting evaluation procedures and benchmark-score sources. Additional details cover fair SalEMA scoring, ACLNet evaluation, and prior-result provenance.

  • 8 MIT300 Probabilistic Benchmark Results: Metric-specific adaptation is omitted to preserve comparability, although s-AUC, CC, and KLD require different saliency-map transformations.The passage notes division by the center-bias map for s-AUC and smoothing for CC and KLD.
  • 9.1 Scoring SalEMA with Training Setting (vi): SalEMA is fine-tuned with setting (vi), combining DHF1K, Hollywood-2, UCF Sports, and SALICON before scoring benchmark validation and test sets.SALICON images are treated as single-frame videos, using the authors’ official implementation and default settings for SalEMA30.pt.
  • 9.2 Scoring ACLNet on SALICON: ACLNet is evaluated on SALICON validation using either its auxiliary image-saliency output or LSTM output, with the auxiliary output reported because it scores better on all metrics.The SALICON test set is held out for benchmarking.
  • 9.3 Sources of Other Benchmark Scores: Previous video-model scores on DHF1K, UCF-Sports, and Hollywood-2 are taken from reference.The passage identifies as the source for these prior video-saliency benchmark scores.
  • 9.3 Sources of Other Benchmark Scores: Previous image-model scores on SALICON and MIT300 are taken from the respective papers.These reported scores provide the comparison values for the image-saliency benchmarks.
  • 9.4 Generating MIT300 predictions: 10-fold cross validation on MIT1003 is used for fine-tuning before MIT300 prediction generation, which averages the log-probabilities of the 10 fine-tuned models.This procedure follows the benchmark authors’ recommendation and is similar to reference.
Loading 2003.05477v3…