Source-linked AI summary
Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos
Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, Min Sun
TL;DR
360° video saliency prediction must support viewpoint guidance despite limited scalable supervision and distortions or boundaries introduced by common representations. The paper proposes a weakly supervised spatial-temporal network with Cube Padding and evaluates it on Wild-360, outperforming state-of-the-art methods in speed and quality.
Problem
360° saliency prediction requires scalable supervision and representations that avoid distortion and image boundaries, while existing approaches rely on annotated maps or separate perspective images.
Method
The paper introduces a weakly supervised spatial-temporal saliency network using Cube Padding across six perspective-projected cube faces, with a Wild-360 dataset for evaluation.
Results
The method outperforms state-of-the-art methods and all baselines in both speed and quality on the Wild-360 evaluation.
Takeaways & Limitations
Cube Padding provides a simple modification to existing CNN layers that avoids image boundaries and recognizes object patterns across cube faces.
Takeaways & Limitations
The temporal model assumes that steady, low-motion pixels should receive lower saliency scores than changing patches.
Abstract
from arXiv · showhide
Automatic saliency prediction in 360° videos is critical for viewpoint guidance applications (e.g., Facebook 360 Guide). We propose a spatial-temporal network which is (1) weakly-supervised trained and (2) tailor-made for 360° viewing sphere. Note that most existing methods are less scalable since they rely on annotated saliency map for training. Most importantly, they convert 360° sphere to 2D images (e.g., a single equirectangular image or multiple separate Normal Field-of-View (NFoV) images) which introduces distortion and image boundaries. In contrast, we propose a simple and effective Cube Padding (CP) technique as follows. Firstly, we render the 360° view on six faces of a cube using perspective projection. Thus, it introduces very little distortion. Then, we concatenate all six faces while utilizing the connectivity between faces on the cube for image padding (i.e., Cube Padding) in convolution, pooling, convolutional LSTM layers. In this way, CP introduces no image boundary while being applicable to almost all Convolutional Neural Network (CNN) structures. To evaluate our method, we propose Wild-360, a new 360° video saliency dataset, containing challenging videos with saliency heatmap annotations. In experiments, our method outperforms baseline methods in both speed and quality.
1. Introduction
The paper targets scalable saliency prediction for 360° videos by avoiding strong 360° supervision and projection artifacts. It combines weak supervision, Cube Padding, and the Wild-360 dataset, reporting higher speed and quality than baselines.
- Motivation: 360° saliency prediction supports viewpoint guidance, but existing approaches require spatial-temporal saliency maps or related attention information.These applications include optimal NFoV trajectories, VR visual guidance, narrated-video guidance, and highlight generation.
- Challenges: Existing image and video saliency datasets are poorly suited to 360° viewing-angle variation, while proprietary viewing histories are not publicly available.The paper describes this as a chicken-and-egg problem because attention guidance is needed before viewing data can be broadly collected.
- Challenges: Equirectangular images introduce boundaries and severe top-bottom distortion, whereas separate or overlapping perspective images trade distortion for boundaries, computation, or both.The paper positions a simpler modification to existing CNNs as an alternative to spherical convolution.
- Method: The proposed spatial-temporal network is weakly supervised without 360° videos and uses a static model with ConvLSTM temporal aggregation.The static model uses monocular image-level supervision, while the temporal model aggregates static features across timesteps.
- Method: Cube Padding projects the sphere onto six cube faces, concatenates them, and uses face connectivity for padding in convolution, pooling, and convolutional LSTM layers.This design introduces little distortion, no image boundary, and remains applicable to existing CNN layers.
- Evaluation: Wild-360 is a new challenging 360° video saliency dataset, and experiments report that the method outperforms baselines in both speed and quality.The contribution summary states that one-third of the dataset has per-frame saliency heatmaps aggregated from viewers’ trajectories.
2. Related work
The related work spans saliency prediction, weakly supervised or unsupervised localization, 360° video techniques, and datasets. The paper distinguishes its approach from prior methods that rely on strong supervision or suffer from projection boundaries and distortion.
- Saliency map prediction: Prior saliency work detects salient regions in images and can use spatial heatmaps to learn or predict temporal saliency.Other work couples image-level classification with pixel-level generation to estimate saliency.
- Unsupervised localization: Unsupervised localization commonly uses reconstruction losses, while related video-segmentation models use recurrent architectures to combine visual information over time.The paper uses ConvLSTM to combine spatial and temporal, including motion, information sequentially.
- 360° video: Earlier 360° saliency work converts spherical patches into perspective images, detects saliency with a SALICON-pretrained model, and refines the result using patch location.The paper notes that such approaches have image-boundary defects and require strong supervision.
- Dataset: Wild-360 is presented alongside established saliency resources such as MIT300 and SALICON, with AUC-Judd and AUC-Borji used as evaluation metrics.The dataset discussion frames Wild-360 as an effort to establish a collection for 360° video analysis.
3. Our method
The method projects 360° views onto cubemap faces and uses Cube Padding to preserve cross-face spatial context throughout static and temporal saliency prediction. A ConvLSTM aggregates saliency features over time, while self-supervised temporal losses refine the outputs.
- Projection and overall pipeline: The system projects equirectangular frames into six-face cubemaps, processes them with CNN layers using Cube Padding, and converts predictions back to equirectangular saliency maps.The static pipeline uses cubemap projection, CP-enhanced CNN feature extraction, inverse projection, and upsampling.
- Cube Padding: Cube Padding copies adjacent regions from neighboring cube faces into each target face, allowing convolutional receptive fields to cross face boundaries.For the front face, neighboring information comes from the right, top, left, and down faces rather than zero values.
- Cube Padding: Across multiple CP-enabled layers, receptive fields expand over nearby faces, enabling object patterns to remain recognizable across cubemap boundaries.Feature visualizations contrast continuous CP responses with vanished responses near boundaries under Zero Padding.
- Static model: The static model forms saliency features by convolving the final feature map with classification-layer weights, then selects the maximum class response at each pixel.The resulting cube-face saliency map is inverse-projected and upsampled to the final equirectangular output.
- Temporal model: A ConvLSTM aggregates static saliency features across timesteps, using CP to retain spatial connectivity between cube faces in temporal processing.Its hidden representation is converted into saliency maps using the same maximum-over-class operation as the static model.
- Temporal model: The temporal model jointly uses reconstruction, smoothness, and motion-masking losses to impose temporal constraints on predicted saliency maps.The reconstruction term compares the current output with a motion-warped previous output; smoothness limits large changes, and motion masking suppresses responses below a motion threshold.
4. Dataset
Wild-360 is introduced as a publicly benchmarkable 360° video saliency dataset designed to capture diverse objects, dynamic content, and viewing angles while reducing center bias.
- Wild-360 contains 85 360° video clips totaling about 55k frames, with 60 clips for training and 25 for testing.
- The clips are trimmed from 45 YouTube videos selected across Nature, Wildlife, and Animals keywords.
- The dataset targets multiple salient objects, diverse categories, and dynamic content appearing across arbitrary viewing angles, including polar and border regions.
- Testing videos are rotated in longitude and latitude to prevent center bias in the ground-truth saliency.
- Wild-360 uses the HumanEdit interface, where 30 labelers record attention trajectories based on intuition while viewing the global sphere.
5. Experiments
Experiments evaluate saliency accuracy, inference speed, qualitative quality, NFoV piloting, and human preferences. The proposed static and temporal models generally outperform the baselines across these evaluations.
- Evaluation setup: The evaluation compares saliency accuracy and speed against multiple baseline methods using CC, AUC-J, and AUC-B metrics.The experiments include EQUI, Cubemap, Overlap, and ConvLSTM-based alternatives.
- Computational efficiency: Ours Static is slower than Cubemap but faster than Overlap and EQUI across tested resolutions, while Ours is significantly faster than EQUI+ConvLSTM.Resolutions range from 1920 to 3840, and speed is measured in FPS on an NVIDIA Tesla M40 GPU.
- Saliency comparison: The temporal model is best on all but one saliency metric, and ConvLSTM generally improves both the EQUI and proposed models.The temporal model also produces smoother maps and better captures salient regions near image boundaries and at the sphere’s top and bottom.
- Qualitative results: Qualitative examples show the temporal model overcoming distortion, image boundaries, and temporal smoothness issues more effectively than other methods.The visualizations compare consecutive frames, ground truth, raw videos, zoomed regions, and salient NFoVs.
- NFoV piloting: NFoV trajectories generated from the proposed saliency features capture salient viewpoints better than equirectangular processing.The piloting demonstration uses Wild-360 and Drone videos and links salient viewpoints with AUTOCAM.
- Human evaluation: Human evaluation finds Ours Static statistically superior to EQUI and Cubemap, while Ours significantly outperforms Ours Static and is comparable to ground truth.Sixteen viewers evaluated ten clips per comparison; reported p-values are below 0.05, below 0.01, and 0.13, respectively.
6. Conclusion
The paper presents a weakly supervised spatial-temporal network for 360° saliency prediction, combining Cube Padding with the Wild-360 dataset. It reports improved speed and quality over state-of-the-art methods.
- 6. Conclusion: The proposed network is trained without 360° supervision and uses Cube Padding tailored to the 360° viewing sphere.The conclusion describes Cube Padding as simple and effective within the proposed spatial-temporal network.
- 6. Conclusion: On the newly collected Wild-360 dataset, the method outperforms state-of-the-art methods in both speed and quality.Wild-360 contains challenging videos with saliency heatmap annotations.