Source-linked AI summary
SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes
Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto, Noel E. O'Connor
TL;DR
Traditional saliency maps omit fixation order and duration, leaving temporal information unrepresented for 360° scanpath prediction. SaltiNet predicts spatiotemporal saliency volumes and samples scanpaths from them; it earned the best scanpath solution at the Salient360! challenge, though sampling accuracy depends on strategy.
Problem
Traditional saliency maps encode spatial fixation likelihood but not scan order or duration, while 360° viewers choose their observation angle.
Source: §1 · p. 1Method
Results
SaltiNet was awarded the best scanpath solution at the 2017 Salient360! challenge, and challenge results placed it above the two other participants.
Source: §1 · p. 1, §5.4 · p. 5Takeaways & Limitations
Sampling from saliency volumes can produce realistic scanpaths, and accuracy depends greatly on the sampling strategy.
Source: §6 · p. 5Fixation probability is not conditioned on previous fixations, and scanpath length and fixation duration are treated as independent despite their dependence.
Source: §6 · p. 5
Abstract
from arXiv · showhide
We introduce SaltiNet, a deep neural network for scanpath prediction trained on 360-degree images. The model is based on a temporal-aware novel representation of saliency information named the saliency volume. The first part of the network consists of a model trained to generate saliency volumes, whose parameters are fit by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency volumes. Sampling strategies over these volumes are used to generate scanpaths over the 360-degree images. Our experiments show the advantages of using saliency volumes, and how they can be used for related tasks. Our source code and trained models available at https://github.com/massens/saliency-360salient-2017.
1. Motivation
Traditional saliency maps encode where people look but not fixation order or duration, a limitation relevant to viewing 360° scenes. SaltiNet introduces temporal saliency volumes and a network-plus-sampling approach to predict scanpaths.
1. Motivation
- Traditional saliency maps encode pixel-level fixation probability but omit the order of scanned pixels and fixation duration.Source: §1 · p. 1
- For 360° images, scanpath prediction matters because viewers choose their observation angle, with potential relevance to VR/AR encoding and rendering.Source: §1 · p. 1
- The paper introduces saliency volumes to capture the temporal nature of eye-gaze scanpaths.Source: §1 · p. 1
- SaltiNet predicts saliency volumes with a deep network and generates scanpaths using a sampling strategy.Source: §1 · p. 1
- SaltiNet was awarded the best scanpath solution at the 2017 Salient360! challenge.Source: §1 · p. 1
2. Related Work
Prior saliency work progressed from handcrafted visual features to deep networks and contextual models, while scanpath prediction remained less developed. Earlier scanpath studies addressed fixation tendencies and 360° video viewing, rather than the paper’s 360° image setting.
2.1. Saliency prediction
- Earlier saliency models combined low-level features with higher-level semantic cues after low-level-only approaches often failed to match human saccades.Judd et al. used image features with a linear SVM; Borji combined low-level and top-down features with learned classifiers.Source: §2.1 · p. 2
- Deep learning advanced saliency prediction, with models reusing image-recognition features, designing saliency-specific architectures, or applying adversarial and attentive mechanisms.Examples include DeepGaze [17], DeepGaze 2, SalNet [24], SalGAN [23], and SAM [6].Source: §2.1 · p. 2
- Saliency research also explored global and scene context [21] and differentiable metric-based loss functions, including NSS, CC, SIM, and KL divergence.Source: §2.1 · p. 2
2.2. Scanpath prediction
- Scanpath prediction had seen less progress than saliency maps; prior work reported face-fixation patterns and used object proposals with an RNN to select viewing directions in 360° video [9].Source: §2.2 · p. 2
3. Architecture
SaltiNet predicts a spatiotemporal saliency volume with a convolutional network, then samples fixation locations and scanpath attributes from learned distributions. The volume can also yield standard saliency representations, while direct LSTM scanpath prediction regressed toward the image center.
3. Architecture
- SaltiNet’s central component is a DCNN that predicts a saliency volume from an input image.Source: §3 · p. 3
3.1. Saliency Volumes
- Saliency volumes represent image width, height, and time; they are built by quantizing fixation timestamps, placing fixation events in a binary volume, and Gaussian convolution followed by per-slice normalization.Source: §3.1 · p. 3
- Temporal slices can be summed and normalized into saliency maps, weighted and summed into temporally weighted maps, or sampled to extract scanpaths.The representation is therefore used to derive both spatial summaries and temporal scanpaths.Source: §3.1 · p. 3
3.2. Convolutional Neural Network
- The CNN adapts filters used for flat saliency maps; its 10 layers contain 25.8 million parameters, and its first three layers are initialized from VGG-16.Inputs are resized to 300 × 600 and the output volume is 12 × 300 × 600.Source: §3.2 · p. 3
3.3. Scan-path sampling
- Scanpath sampling draws fixation count and duration from training-data distributions and fixation locations from the corresponding predicted temporal slices.A direct LSTM scanpath attempt regressed to the image center; adversarial training was proposed for future work.Source: §3.3 · p. 3
4. Training
SaltiNet was trained on downsampled saliency volumes, with a held-out validation set and binary cross-entropy optimization. Validation monitoring and L2 regularization were used during training.
Training data
- 36 of 40 Salient360 images were used for training and 4 for validation; images and volumes were downsampled to 600 × 300.Volumes were normalized to [0, 1] and generated with a multivariate Gaussian kernel with bandwidths for time, height, and width.Source: §4 · p. 4
Optimization
- The CNN used stochastic gradient descent with binary cross-entropy, batch size 1, and 90 training epochs.The loss compares ground-truth and predicted saliency-map values.Source: §4 · p. 4
Training monitoring
- Validation performance was tracked, and L2 weight regularization was used to address overfitting.Training took approximately two hours on an NVIDIA GTX Titan X, with learning rate α = 0.001.Source: §4 · p. 4
5. Experiments
The experiments evaluate sampling choices and scanpath prediction using a 360-degree eye-tracking dataset and a modified scanpath-similarity metric. SaltiNet outperformed the other Salient360! Challenge participants, while results also depended on the sampling strategy.
5.1. Datasets
- Training progressed from SALICON saliency-map prediction to iSUN saliency-volume prediction, then fine-tuning on 60 HMD-viewed images with eye-tracking data.The 360-degree dataset recordings involved 40–42 observers, each viewing images for 25 seconds.Source: §5.1 · p. 4
5.2. Metric
- The evaluation used a Jarodzka-algorithm variation with equirectangular distances and one-to-one Hungarian matching of 40 generated scanpaths to ground truth.The metric was proposed by the 360 saliency dataset authors.Source: §5.2 · p. 4
5.3. Sampling strategies
- Fixation counts and durations were sampled from empirical training-set distributions when generating scanpaths.Figure 4 shows fixation counts at top and fixation durations at bottom.Source: §5.3 · p. 4, Figure 4 · p. 5
- Sampling strategy (2), which biases each fixation toward the previous fixation, performed best among the three strategies.The comparison reports lower values as better.Source: §5.3 · pp. 4–5, Table 1 · p. 5
5.4. Results
- The sampling-strategy evaluation also compared accuracy with random-fixation and ground-truth-fixation outputs.Table 1 presents the impact of the different strategies.Source: §5.4 · p. 5
- SaltiNet showed superior performance to the two other Salient360! Challenge participants.Table 2 compares three challenge submissions, with lower values indicating better performance.Source: Table 2 · p. 5, §5.4 · p. 5
6. Conclusions
SaltiNet predicts scanpaths on 360-degree images using a temporal-aware saliency representation. The authors report realistic scanpaths with accuracy sensitive to sampling strategy, alongside limitations in modeling dependencies among scanpath variables.
Contributions
- SaltiNet predicts scanpaths on 360-degree images using saliency volumes.The volumes can also generate scanpaths, saliency maps, and temporally weighted saliency maps.Source: §6 · p. 5
Findings
- Experiments found that sampling from saliency volumes can produce realistic scanpaths, with accuracy depending greatly on sampling strategy.Source: §6 · p. 5
Limitations and future work
- The model treats fixation probability as unconditioned on previous fixations and scanpath length and fixation duration as independent random variables, although the three parameters are not independently distributed.The authors state that this limits accurate representation of the relationship and propose end-to-end scanpath prediction as future work.Source: §6 · p. 5
Reproducibility
- The source code and trained models are available online for reproducing the results.Source: §6 · p. 5