PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Yuhang Song, Mai Xu, Jianyi Wang, Minglang Qiao, Liangyu Huo, Zulin Wang

arXiv:1710.10755v5cs.CVcs.LG

TL;DR

Panoramic-video attention modeling requires predicting head movements that determine viewers’ fields of view, but prior HM prediction evidence is limited. The paper collects a multi-subject database and develops DRL-based offline and online predictors; experiments report effectiveness in both tasks and improved online performance when using the learned offline model.

  • Problem

    HM prediction is important for modeling attention in panoramic video, yet few approaches address it and no offline work predicts multiple subjects’ HM positions.

  • Method

    The paper builds a database and uses DRL to imitate human HM scanpaths, producing offline HM maps and online next-position predictions from video and observed scanpaths.

  • Results

    Both offline-DHP and online-DHP are reported as superior to conventional approaches, while incorporating the offline DRL network raises average MO from 0.50 to 0.75 across 15 sequences.

  • Takeaways & Limitations

    The learned offline-DHP model benefits online HM prediction, while HM maps represent consistent HM positions across humans.

  • Takeaways & Limitations

    The work mainly predicts HM positions as a first step, leaving eye-fixation prediction within FoV regions for future work.

Abstract

from arXiv · show

Panoramic video provides immersive and interactive experience by enabling humans to control the field of view (FoV) through head movement (HM). Thus, HM plays a key role in modeling human attention on panoramic video. This paper establishes a database collecting subjects' HM in panoramic video sequences. From this database, we find that the HM data are highly consistent across subjects. Furthermore, we find that deep reinforcement learning (DRL) can be applied to predict HM positions, via maximizing the reward of imitating human HM scanpaths through the agent's actions. Based on our findings, we propose a DRL-based HM prediction (DHP) approach with offline and online versions, called offline-DHP and online-DHP. In offline-DHP, multiple DRL workflows are run to determine potential HM positions at each panoramic frame. Then, a heat map of the potential HM positions, named the HM map, is generated as the output of offline-DHP. In online-DHP, the next HM position of one subject is estimated given the currently observed HM position, which is achieved by developing a DRL algorithm upon the learned offline-DHP model. Finally, the experiments validate that our approach is effective in both offline and online prediction of HM positions for panoramic video, and that the learned offline-DHP model can improve the performance of online-DHP.

1 INTRODUCTION

The paper frames HM prediction as a foundational problem for modeling attention in panoramic video and addresses limited prior work with a DRL-based approach. It introduces a database, offline HM-map prediction, and online next-position prediction.

  • HM determines the FoV viewed in panoramic video, making HM prediction a first step toward modeling human attention.
  • Few approaches had modeled HM in panoramic video, while existing online work relied heavily on tracking one salient object.
  • DRL optimizes rewards measuring how closely agent actions imitate human HM scanpaths, and multiple workflows combine predicted positions into HM maps.
  • The database contains HM positions from 58 subjects across 76 panoramic video sequences and reveals high cross-subject consistency.
  • Offline-DHP predicts consistent HM positions for multiple subjects by generating an HM map for each panoramic frame.
  • Online-DHP predicts one subject’s HM position at the next frame from video content and the HM scanpath observed through the current frame.

2 RELATED WORK

Prior work addresses visual attention and virtual cinematography mainly through 2D saliency detection or object-focused panoramic-video viewing. The paper distinguishes its HM prediction task by modeling spherical, multi-subject HM behavior without depending on an object-of-interest detector.

  • 2.1 Saliency detection: Pano2Vid generates a potential FoV at each panoramic-video frame but does not model HM maps for multiple subjects.
  • 2.1 Saliency detection: Existing saliency research includes heuristic feature combinations, earth mover’s distance, Boolean-map models, and deep neural networks for 2D video.
  • 2.1 Saliency detection: 2D saliency detection assumes viewers can see an entire planar frame, unlike panoramic viewing with a limited FoV on a sphere.
  • 2.1 Saliency detection: The paper predicts panoramic HM maps by estimating HM scanpaths of several agents using a DRL approach driven by panoramic-video content.
  • 2.2 Virtual cinematography: Virtual cinematography seeks natural fields of view by matching human HM positions, with prior methods commonly detecting or tracking salient objects.
  • 2.2 Virtual cinematography: Unlike detector-dependent approaches, online-DHP uses an offline-learned model that encodes HM-related content rather than detecting an object of interest.

3 DATABASE ESTABLISHMENT AND FINDINGS

The PVS-HM database records head movements and eye fixations from 58 subjects viewing 76 diverse panoramic video sequences. Analysis finds strong cross-subject consistency in HM positions, scanpath magnitudes, and directions.

  • Database: The PVS-HM database contains HM and eye-fixation data from 58 subjects across 76 panoramic video sequences.Sequences span diverse content, use 3K–8K resolution, and last 10–80 seconds, averaging 26.9 seconds.
  • Database: Subjects viewed all sequences through an HTC Vive while freely turning on a swivel chair, enabling access to all panoramic regions.HM positions were recorded frame by frame in longitude and latitude coordinates.
  • Findings: The database analysis identifies a front-center bias and high consistency across subjects in HM positions, scanpath magnitudes, and directions.These findings support modeling shared patterns of panoramic-video viewing behavior.
  • Findings: Almost 50% of subjects are consistent in one HM scanpath direction, while over 85% are consistent in three directions.Directions are represented using 8 uniformly quantized directions.

4 OFFLINE-DHP APPROACH

Offline-DHP predicts multiple plausible HM scanpaths from panoramic video using shared DRL workflows, then combines their positions into per-frame HM maps. Its rewards encourage agreement with subjects’ scanpath directions, magnitudes, and positions.

  • Framework: Offline-DHP runs multiple DRL workflows on panoramic video to generate potential HM positions and per-frame HM maps.Each workflow predicts a scanpath, and the resulting positions are integrated and smoothed with a 2D Gaussian filter.
  • HM map: The HM map represents the probability of each pixel being the HM position, obtained by convolving predicted positions with a 2D Gaussian filter.The number of predicted positions N is set equal to the number of subjects M at each frame.
  • DRL training: Multiple workflows share one global DRL model and interact asynchronously with environments formed by FoV observations and HM scanpath rewards.The network uses convolutional layers and an LSTM to extract spatial and temporal features.
  • Reward estimation: The reward models similarity between predicted and ground-truth scanpaths using direction, magnitude, and position-related terms.Gaussian distributions measure differences, while validity factors restrict magnitude similarity to sufficiently matching positions and directions.

5 ONLINE-DHP APPROACH

Online-DHP predicts one viewer’s next HM position from observed scanpath history and panoramic video content. It first adapts an offline-DHP-initialized DRL network, then uses the trained network to generate the next scanpath action and position.

  • Framework: Online-DHP predicts a viewer’s HM position at frame t + 1 from HM positions through frame t and panoramic video content.Its input includes the viewer’s previous scanpath and frames, while its output is the next HM position.
  • Stages: Online-DHP has separate training and prediction stages for adapting the DRL network and generating the next HM scanpath action.The online network is initialized with offline-DHP parameters and fine-tuned using the viewer’s observed scanpath.
  • Training: At each incoming frame, episodes update the DRL network using predicted actions, FoV observations, rewards, and stored experiences.The action direction selects one of 8 discrete directions, while the magnitude is scalar-valued.
  • Training: Training stops after a maximum of E episodes or when mean overlap exceeds threshold thMO.Mean overlap ranges from 0 to 1, with larger values indicating more precise prediction.
  • Prediction: During prediction, the DRL network updates LSTM features from observed FoVs and produces scanpath direction and magnitude for the next HM position.The next position is computed from the current ground-truth HM position and the estimated scanpath.

6 EXPERIMENTAL RESULTS

Experiments evaluate offline-DHP and online-DHP through ablations and comparisons on panoramic-video HM prediction. Offline-DHP outperforms competing HM-map approaches, while online-DHP improves over baselines and Deep 360 Pilot.

  • Experimental settings: Experiments use 61 training sequences and 15 test sequences from the 76-sequence PVS-HM database; offline-DHP uses 58 workflows.The workflow count matches the 58 subjects in the database.
  • Ablation experiments: Incorporating the trained offline-DHP network increases online-DHP's average MO from 0.50 to 0.75 across all 15 sequences.The offline network is used as prior knowledge for online prediction.
  • Offline-DHP: Offline-DHP predicts HM maps more accurately than OBDL, BMS, SALICON, and the FCB baseline on average CC and NSS.With FCB integrated, average CC improves by 0.242, 0.198, and 0.134 over OBDL, BMS, and SALICON; average NSS improves by 1.245, 1.087, and 0.856, respectively.
  • Offline-DHP: Offline-DHP also achieves higher average shuffled-AUC than the competing approaches and ranks first on 6 of 15 test sequences.The authors attribute this partly to shuffled-AUC removing FCB influence and to offline-DHP producing sparser HM maps closer to ground truth.
  • Offline-DHP: The visualized HM maps from offline-DHP are considerably closer to ground-truth HM maps than those generated by BMS, OBDL, and SALICON.Figure 8 compares ground truth from all 58 subjects with predictions from the four approaches.
  • Online-DHP: Online-DHP outperforms two baselines and Deep 360 Pilot, increasing average MO over Deep 360 Pilot by 0.35 across the 15 test sequences.Online-DHP also outperforms Deep 360 Pilot on almost all sequences and achieves 0.67 MO versus 0.08 on Waterfall.

7 CONCLUSION

The paper introduces DHP for panoramic-video HM prediction, combining a database finding of cross-human consistency with offline and online DRL prediction. It also identifies HM prediction as an initial step toward broader panoramic-video attention modeling and notes potential coding applications.

  • The PVS-HM database contains HM data from 58 subjects viewing 76 panoramic sequences, with HM positions highly consistent across humans.
  • Offline-DHP uses DRL to imitate human panoramic-video viewing and integrates scanpaths from multiple agents to produce HM maps.
  • Online-DHP predicts one subject’s HM positions at incoming frames from previous HM scanpaths and current video content.
  • The paper positions HM prediction as a first step toward modeling panoramic-video attention, with future work targeting eye fixations within FoV regions.
  • Offline-DHP may reduce panoramic-video perceptual redundancy and thereby save coding bit-rates.

APPENDIX A ANALYSIS OF FCB COMBINED IN HM MAPS

The appendix combines a front-center bias feature with predicted HM maps using weighted channels. Least-squares fitting over the PVS-HM training data selects the combination parameters, and the best reported setting reaches a CC of 0.70.

  • The appendix models front-center bias with a 2D Gaussian feature channel for each panoramic-video frame.The feature uses the front center position in GCS and a standard deviation σf.
  • The combined HM map integrates the front-center-bias map Hf with the predicted HM map Ht using weights w1 and w2.
  • Least-squares fitting of CC over all PVS-HM training data obtains σf = 21.1°, w1 = 0.48, and w2 = 0.52.
  • CC results range from 0.44 to 0.70 as w1 increases from 0 to 1, with the maximum at w1 = 0.48 and σf = 21.1°.
Loading 1710.10755v5…