Source-linked AI summary
SalNet360: Saliency Maps for omni-directional images with CNN
Rafael Monroy, Sebastian Lutz, Tejo Chalasani, Aljosa Smolic
TL;DR
Traditional 2D saliency prediction does not directly accommodate omnidirectional images because their projections contain heavy distortions and different viewing biases. The paper extends CNN-based saliency prediction with an end-to-end ODI pipeline using undistorted patches and spherical coordinates, and reports evidence that these design choices improve saliency-map accuracy against ground truth.
Problem
Traditional 2D saliency prediction cannot be directly used on ODIs because projected images contain heavy distortions and different observed biases.
Method
The paper presents an end-to-end CNN extension that predicts ODI saliency from undistorted patches and per-pixel spherical coordinates.
Results
The paper provides evidence that dividing ODIs into patches and adding spherical coordinates to a Base CNN considerably improves saliency-map estimation.
Takeaways & Limitations
The proposed pipeline makes CNN-based saliency prediction more suitable for omnidirectional images by addressing projection distortions before estimating saliency.
Takeaways & Limitations
The authors identify remaining visible artifacts after Gaussian blurring and note that the network used a relatively simple Euclidean Loss function, leaving room for improvement.
Abstract
from arXiv · showhide
The prediction of Visual Attention data from any kind of media is of valuable use to content creators and used to efficiently drive encoding algorithms. With the current trend in the Virtual Reality (VR) field, adapting known techniques to this new kind of media is starting to gain momentum. In this paper, we present an architectural extension to any Convolutional Neural Network (CNN) to fine-tune traditional 2D saliency prediction to Omnidirectional Images (ODIs) in an end-to-end manner. We show that each step in the proposed pipeline works towards making the generated saliency map more accurate with respect to ground truth data.
1. Introduction
Visual-attention prediction identifies where people look and can support compression and segmentation, but traditional 2D saliency methods do not transfer directly to ODIs because projections distort images and viewing biases differ. The paper addresses this gap with an end-to-end CNN extension using undistorted patches and spherical coordinates.
- Motivation: Visual attention prediction estimates the regions people are most likely to look at in an image.Ground-truth saliency maps are built from eye-tracking and headset inertial-measurement data collected while subjects view images.
- Motivation: Saliency maps can guide compression and segmentation algorithms by identifying the most observed image regions.
- Problem: Traditional 2D saliency techniques cannot be directly applied to ODIs because projections introduce heavy distortions and viewing biases differ.
- Evaluation setting: The paper follows Salient360! challenge conditions, training the proposed CNN on 40 images and evaluating it on 25 test images.
- Approach: The proposed contributions subdivide ODIs into undistorted patches and provide the CNN with spherical coordinates for each pixel.
2. Previous work
Earlier saliency systems combined handcrafted features, while later CNN-based approaches learned features and location-dependent patterns from data. ODI-specific CNN research remained limited, motivating methods that account for spherical imagery and projection distortions.
- Traditional saliency: Early saliency models manually extracted low-level, global, face, and location-bias features before combining them into saliency maps.
- ODI saliency: ODI saliency methods are needed because ODIs represent a sphere, so traditional 2D-image models cannot be directly used.
- CNN saliency: CNN-based methods became prominent in saliency prediction, using convolutional features, multi-resolution inputs, end-to-end training, and location-biased layers.
- CNN saliency: Some CNN approaches used pretrained features without fine-tuning, while others combined multiscale convolutional and handcrafted features.
- ODI saliency: The published research applying CNNs to ODIs was described as very limited, with prior work using translated ODI versions as input to a CNN trained for traditional 2D images.
3. Method
The method converts an ODI into six low-distortion patches, processes each patch with a CNN together with spherical coordinates, and reprojects the outputs into one equirectangular saliency map. Post-processing uses nearest-neighbour forward projection and Gaussian smoothing to fill holes and smooth the result.
- Pipeline: The pipeline splits each ODI into six patches, processes them independently with a CNN, and combines their outputs into one saliency map.
- Pre-processing: The patches use approximately 90-degree fields of view to keep distortions low while covering the sphere with six views.
- Pre-processing: Each patch receives per-pixel spherical coordinates, which are used to locate corresponding pixels in the ODI.
- Pre-processing: During saliency computation, two fixed views target the nadir and zenith, while four rotated horizontal views cover the equatorial band.
- Post-processing: The CNN generates a saliency map for each patch, and per-pixel spherical coordinates reproject these maps into the equirectangular ODI.
- Post-processing: Nearest-neighbour forward projection is followed by a 64-pixel Gaussian filter to fill holes and smooth the combined map.The paper states that this kernel size gave the best correlation with ground truth among the tested processing choices.
4. SalNet360
SalNet360 extends a CNN trained on traditional 2D images to omnidirectional images by processing undistorted patches and adding per-pixel spherical coordinates. The architecture combines a Base CNN with a Saliency Refinement stage and is trained in two stages.
- Motivation: ODIs are large, and limited training data makes directly training a CNN impractical.The authors address the 40-image dataset by generating 100 paired patches per ODI.
- Pipeline: The method divides each ODI into undistorted patches, predicts their saliency maps separately, and recombines the results.The patch-based design avoids heavily down-scaling the entire ODI during inference.
- Pipeline: SalNet360 adds each pixel’s spherical coordinates to the Base CNN saliency output for Saliency Refinement.The refinement stage can adjust salient regions according to their placement in the ODI, including the horizon bias.
- Training: ReLU activations and Euclidean Loss yielded the best metrics on the dataset among the tested activation and loss configurations.The full architecture uses a Saliency Refinement module after the Base CNN.
5. Results
The experiments compare a Base CNN, patch processing, and patch processing with spherical coordinates using four saliency metrics. Performance improves across most metrics with patches and considerably across all metrics after adding spherical coordinates, although visual artefacts remain.
- Performance measures: The evaluation uses KL, CC, NSS, and AUC, separating distribution-based metrics from location-based metrics.KL and CC are distribution-based; NSS and AUC are location-based.
- Experiments: The Base CNN applied to the entire ODI produces relatively poor results compared with the later experimental scenarios.Table 2 reports average KL, CC, NSS, and AUC over 25 test images.
- Experiments: Using individually processed patches before recombination improves results in most metrics over the Base CNN.The second scenario also finds salient areas that overlap ground-truth salient regions more closely in the visual example.
- Experiments: Adding spherical coordinates after patch processing considerably improves results in all four metrics.The third scenario covers ground-truth salient areas and removes incorrectly labelled salient regions introduced by the patch-only scenario.
- Challenges: The spherical-coordinate scenario covers salient regions too generously, leaving the amount of predicted salient area as a main issue.The paper also reports lattice-like merging artefacts that can negatively affect KL and CC scores.
- Salient360! Grand Challenge: The approach was submitted to the Salient360! head+eye category, where 16 individual submissions were evaluated and Table 3 lists the top five performers.The challenge used head- and eye-tracking ground truth data for omnidirectional images.
6. Conclusions
The paper presents an end-to-end CNN tailored to ODI saliency prediction, combining undistorted patches with per-pixel spherical coordinates. Results indicate that these adaptations improve prediction, while recombination artefacts, model simplicity, and limited data constrain performance.
- Dividing ODIs into patches and recombining their predictions improves results considerably across all reported metrics.The conclusion links patch-based processing to improved omnidirectional saliency results.
- Adding spherical coordinates for each pixel to an existing Base CNN can considerably improve omnidirectional saliency prediction.The coordinates account for object-location biases associated with positions on the sphere.
- The network uses Euclidean loss, minimizing pixel-wise differences between the predicted saliency map and ground truth.The paper notes that custom losses could instead optimize metrics such as KL or CC.
- Recombining patches creates artefacts that are currently mitigated with Gaussian blur, although a more sophisticated process could exploit their recurring locations.The artefacts appear consistently in the same way and location.
- Performance remains constrained by the relatively simple Base CNN and the small amount of available omnidirectional saliency ground-truth data.The authors expect newer CNN advances and more data to improve results.
- The proposed end-to-end CNN is specifically tailored to estimate saliency maps for omnidirectional images.It extends CNN-based saliency prediction beyond traditional 2D-image settings.
- Dividing an ODI into undistorted patches addresses distortions from projected images before saliency-map computation.The paper identifies projected-ODI distortions as a source of discrepancies for CNNs trained on traditional 2D images.
Appendix A Summary of results per image
The appendix reports individual ODI results and identifies the two best and two worst examples for inspection.
- Individual results are reported for each omnidirectional image.
- The two best and two worst results are highlighted in green and red, respectively.