Source-linked AI summary
Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video
Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, Min Sun
TL;DR
Viewing 360° sports video requires continuous angle control, especially as objects of interest move. The paper learns an online deep 360 pilot agent that selects objects and steering angles jointly, combining accuracy, smoothness, and foreground focus objectives. On a five-domain dataset, it reports the best viewing-angle accuracy and transition smoothness against the cited baselines and notes scope limits when its object-salience assumption fails.
Problem
360° sports-video viewers must continuously steer the viewing angle, a cumbersome task that can cause discomfort as foreground objects move.
Method
The paper learns an online policy that detects candidate objects, selects a main object, predicts viewing angles from current and prior observations, and jointly optimizes supervised and policy-gradient objectives.
Results
The agent achieves the best viewing-angle selection accuracy and transition smoothness compared with AUTOCAM and other baselines across the evaluated sports domains.
Takeaways & Limitations
Automatic online piloting can capture foreground sports events while steering the viewing angle smoothly for viewer-oriented 360° video navigation.
Takeaways & Limitations
The algorithm can suffer when domains contain equally salient objects or no objects, and the authors identify reducing ground-truth annotation as future work.
Abstract
from arXiv · showhide
Watching a 360° sports video requires a viewer to continuously select a viewing angle, either through a sequence of mouse clicks or head movements. To relieve the viewer from this "360 piloting" task, we propose "deep 360 pilot" -- a deep learning-based agent for piloting through 360° sports videos automatically. At each frame, the agent observes a panoramic image and has the knowledge of previously selected viewing angles. The task of the agent is to shift the current viewing angle (i.e. action) to the next preferred one (i.e., goal). We propose to directly learn an online policy of the agent from data. We use the policy gradient technique to jointly train our pipeline: by minimizing (1) a regression loss measuring the distance between the selected and ground truth viewing angles, (2) a smoothness loss encouraging smooth transition in viewing angle, and (3) maximizing an expected reward of focusing on a foreground object. To evaluate our method, we build a new 360-Sports video dataset consisting of five sports domains. We train domain-specific agents and achieve the best performance on viewing angle selection accuracy and transition smoothness compared to [51] and other baselines.
1. Introduction
360° sports-video viewing requires continuous viewing-angle selection, which can be cumbersome and uncomfortable as foreground objects move. Deep 360 pilot addresses this with an online deep-learning agent that follows objects while steering smoothly.
- Motivation: 360° video viewers must repeatedly select viewing angles through mouse clicks or headset movements, making sports viewing cumbersome as foreground objects move continuously.The paper also notes that both interaction modes can cause discomfort.
- Task formulation: The paper formulates 360 piloting as an agent observing panoramic frames and steering viewing angles toward successive preferred goals using prior viewing-angle history.This models how a human viewer navigates the video frame by frame.
- Approach: Deep 360 pilot is an online deep-learning agent that navigates 360° sports videos to capture interesting moments while following foreground objects and maintaining smooth transitions.The pipeline detects candidate objects, selects a main object with an RNN, and predicts the preferred viewing angle with a regressor.
- Approach: The jointly trained pipeline combines viewing-angle regression, transition smoothness, and foreground-object reward objectives, using policy gradients for the discrete main-object decision.The regression loss compares selected and ground-truth viewing angles, while the smoothness loss encourages smoother transitions.
- Contributions: The work introduces a 360° sports-video dataset and presents the agent as suitable for online navigation, streaming videos, and foveated VR rendering.The stated contributions include a first human-like online agent and a first 360° sports-video dataset for training and evaluation.
2. Related Work
Prior work addresses video summarization, saliency detection, and virtual cinematography, but 360 piloting specifically requires selecting spatial viewing angles in raw 360° sports video. Deep 360 pilot instead uses an online human-like agent informed by current and previous observations.
- Video summarization: Video summarization methods mainly decide whether to select frames, whereas 360 piloting selects a viewing angle for each panoramic frame.
- Saliency detection: Saliency-detection research has used image appearance, video motion, faces, people, objects, and contextual information to identify salient regions.
- Virtual cinematography: Virtual cinematography commonly operates in simpler virtual environments or restricted wide-field-of-view videos where objects of interest are easier to extract.
- 360° video navigation: Pano2Vid handles raw 360° videos with an offline method, while this work proposes an online human-like agent using present and previous observations.The online design is positioned for streaming videos and human-in-the-loop applications.
3. Our Approach
The approach learns an online policy that uses object-level observations and viewing history to select a main object, refine its viewing angle, and train smooth steering jointly.
- 3.1. Definitions: The online policy uses current and previous observations together with prior viewing angles to choose each steering action.The task is formulated as online viewing-angle selection, with actions applied relative to the previous viewing angle.
- 3.2. Observing in Object Level: Object-level observations combine appearance, location, and motion features, while an RNN aggregates information across frames to control computation growth.The selector uses an RNN followed by a softmax layer to produce a probability distribution over objects.
- 3.3. Focusing on the Main Object: The selector treats main-object identification as classification, and the selected object location provides the action proposal for viewing-angle steering.The discrete object-selection decision is handled with policy gradient training.
- 3.4. Action Refinement: The regressor refines the selector’s proposed action using motion features and recurrent state, then outputs the final steering vector and viewing angle.The resulting viewing angle is computed by adding the predicted steering action to the previous viewing angle.
- 3. Our Approach: The model is organized into detector, selector, and regressor blocks that respectively extract observations, select an object, and regress the viewing angle.Figure 2 visualizes the detector’s boxes, the selected main object, and the final NFoV predicted by the regressor.
- 3.7. Training: Training minimizes viewing-angle error and a smoothness penalty, while policy gradients maximize the expected reward for selecting foreground objects.The selector and regressor are trained jointly because object location affects the regressor output and the selector reward.
4. Sports-360 Dataset
Sports-3602 is a dataset of 342 360° videos spanning five sports domains, selected for dynamic activities and identifiable objects of interest. It includes annotated ground-truth viewing angles for the videos.
- Sports-3602 contains 342 360° videos across basketball, parkour, BMX, skateboarding, and dance.
- The domains were selected for video availability, dynamic activities, and clear human-identifiable objects of interest in most frames.The identifiable-object criterion supports unambiguous ground-truth viewing-angle annotation.
- Dataset construction removed videos with poor resolution or stitching quality and extracted continuous clips without scene transitions.
- Five human annotators labeled the most salient object for VR viewers in each frame of selected video segments.
- The dataset provides video segments with annotated ground-truth viewing angles and uses an 80% training, 20% testing split by video.
5. Experiments
Experiments evaluate deep 360 pilot using overlap accuracy, trajectory smoothness, benchmark comparisons, qualitative examples, and user preferences on Sports-360.
- Evaluation Metrics: MO measures NFoV overlap with ground truth, while MVD measures curvature through consecutive viewing-angle velocity differences.Higher MO indicates greater overlap; lower MVD indicates smoother trajectories.
- Implementation: The detector generates tracked object observations, retaining N = 16 reliable boxes per frame for the selector.Faster R-CNN proposals are linked into long tracklets before selecting reliable boxes.
- Benchmark Experiments: Table 2 reports that “Ours w/o Regressor” achieves the best MO, whereas “Ours” achieves the best MVD and balances both metrics.AUTOCAM’s very low MVD comes from an offline process.
- User Study: Users significantly preferred the proposed method over AUTOCAM and the no-regressor variant across all five domains, with comparability to experts in skateboarding and dance.The reported comparisons use a user study with statistical tests; the method was indistinguishable from humans in skateboarding and dance under the stated tests.
- Benchmark Experiments: 22% MO gain in BMX and at least 3% gain in Dance over AUTOCAM demonstrate stronger viewing-angle overlap across the reported domains.AUTOCAM still achieves significantly lower MVD, while the proposed method achieves lower MVD than its version without a regressor at similar MO.
- Typical Examples: Qualitative examples show smooth trajectories with adequate view selection, while AUTOCAM struggles with fast-moving objects because of five-second glimpse limits and 198 predefined views.The proposed method requires no post-processing step in these examples.
- Conclusion: The method targets domain-specific settings where a most salient object is clear, and may suffer when objects are equally salient or absent.The authors identify reducing ground-truth annotation requirements as future work.
A. Reward Function
The reward function compares each candidate viewing angle with the ground-truth angle using its distance relative to the ground-truth NFoV coverage. Exact alignment receives the maximum reward, while predictions outside that coverage receive a negative reward.
- The reward is defined from the distance between a candidate angle l_t(i) and the ground-truth viewing angle l_t^gt.
- η represents the distance from a viewing-angle center to the corner of its corresponding NFoV.
- When l_t(i) equals l_t^gt, the candidate receives the maximum reward of 1.
- A candidate receives reward -1 when its distance from the ground-truth angle exceeds η and is therefore outside the ground-truth NFoV.
B. Sensitivity Analysis
The sensitivity experiment varies the number of candidate objects and finds that performance remains stable across the tested settings, while the method continues to outperform the baselines.
- Across N = {8, 16, 32}, deep 360 pilot is not sensitive to the number of candidate objects.
- For all three candidate-object counts, deep 360 pilot still outperforms the other baselines.
C. Typical Examples
The typical examples compare deep 360 pilot with several baselines on videos containing different foreground-motion patterns. The examples include fast motion, multiple main objects, and relatively small player movement.
- The comparison includes AUTOCAM, the model without its regressor, RCNN + Motion, and RCNN + BMS.
- The examples cover a BMX video with a fast-moving foreground object, a skateboarding video with two main skateboarders, and a basketball video with relatively small movement.
- Table 4 is identified as reporting performance on [53].
- The basketball example involves a player running from right to left to finish a slam dunk.
D. Human Evaluation Videos
The paper provides a demo video with examples from multiple domains and compares human labels, AUTOCAM, the full method, and the method without its regressor. A user study used NFoV trajectory comparisons because direct NFoV annotation was inefficient.
- The demo contains three examples from different domains and concurrently compares four methods or references: human label, AUTOCAM, ours, and ours without the regressor.
- The review-response text states that it addresses critical reviewer comments.
- Direct NFoV annotation was inefficient because annotators had to watch videos repeatedly with different NFoVs and perform back-and-forth operations.
- The user study instead compared two NFoV trajectories, matching the targeted use case.
E.2. Apply our model on the dataset in AUTOCAM [53].
Because AUTOCAM’s dataset provides training videos in NFOV rather than 360° format, the authors could not train domain-specific agents on it. They instead applied a skateboarding-trained model to 20 360° testing videos spanning four other domains.
- AUTOCAM’s training videos could not support domain-specific training because they were provided in NFOV rather than 360° format.
- The authors applied their skateboarding-trained model to all 20 AUTOCAM testing videos.The videos were downloaded on January 30, 2017 and included ground truths on the project page.
- The 20 testing videos covered hiking, mountain climbing, parade, and soccer, differing from the skateboarding training domain.
E.3. How order consistency are handled when objects disappear/reappear?
The authors report that the RNN handled changing object order empirically, while the supplementary analyses compare candidate-object sensitivity and viewing results across several methods and sports domains.
- Handling object disappearance and reappearance: When objects disappear or reappear, the score-ordered concatenated feature vector changes between frames.The object features are concatenated according to their scores, so changes in object presence alter the resulting vector.
- Handling object disappearance and reappearance: The RNN empirically embedded different vectors from similar scenes into similar points and did not suffer from the changing-vector issue.
- Sensitivity to candidate objects: Table 5 analyzes the sensitivity of performance to the number of candidate objects N across all five domains.
- Qualitative method comparisons: Figures 7 and 8 compare viewing results from multiple methods across BMX, skateboarding, and basketball examples.Figure 7 compares AUTOCAM, the proposed method, and the proposed method without the regressor; Figure 8 compares RCNN + Motion, RCNN + BMS, and the proposed method.