Pano2Vid: Automatic Cinematography for Watching 360$^{\circ}$ Videos

1 Introduction

• 360° video captures the visual world around the camera, while ordinary NFOV video is usually limited to 65°.This wider view allows later replay focused on different parts of a scene, such as different sharks in a shark-cage recording.Source: §1 · p. 1
• Existing viewing options make viewers manage the viewing direction, tolerate a distorted panoramic projection, or wear a headset that can be uncomfortable or impractical for long periods.Across these approaches, viewers may also remain unaware of events elsewhere in the scene.Source: §1 · p. 2
• Pano2Vid selects a virtual NFOV camera’s pose and motion within 360° video to produce natural-looking video without requiring viewers to choose where to look.The task also aims to relieve videographers of consciously directing the camera.Source: §1 · p. 2
• AutoCam scores candidate viewpoints and events using a model learned from human-captured NFOV web video, then selects camera motion with dynamic programming.The approach targets real video from dynamic cameras and learns cinematographic tendencies from data rather than relying on hand-crafted heuristics.Source: §1 · p. 3
• The paper contributes a 360° web-video dataset annotated with human-directed NFOV trajectories and objective and subjective evaluation protocols.These resources are intended to facilitate evaluation of Pano2Vid.Source: §1 · p. 3

Video summarization

• Unlike video summarization, which selects frames or clips over time, Pano2Vid selects where to look at each instant in uncontrolled 360° video.It also aims for natural, human-like NFOV output, not only coverage of important content.Source: Video summarization · p. 3

Camera selection for multi-video summarization

• Unlike multi-video summarization, which chooses among human-directed camera feeds, this task uses one hand-held 360° camera not intentionally aimed at a subject.The distinction is in the input: multiple directed feeds versus a single undirected panoramic feed.Source: Camera selection for multi-video… · pp. 3–4

Video retargeting

• Unlike video retargeting, Pano2Vid must automatically select content from 360° footage that was captured without human-directed content selection.The spatial downsampling is typically much greater than in retargeting.Source: Video retargeting · p. 4

Visual saliency

• Visual saliency tracks where eyes move during free viewing, whereas Pano2Vid targets where videographers consciously point cameras to make presentable video.The paper reports that saliency is not adequate for automatic cinematography.Source: Visual saliency · p. 4

Virtual cinematography

• Unlike virtual cinematography in simpler virtual environments or restricted static wide-FOV settings, this work addresses complex real-world 360° web video from moving amateur cameras.The paper also describes its approach as the first to learn content-based camera control from data rather than use hand-crafted heuristics.Source: Virtual cinematography · p. 4

3.1 Pano2Vid Problem Definition

• Pano2Vid converts a dynamic panoramic 360° video into natural-looking NFOV video, defined here as a 65.5° horizontal view with a 4:3 aspect ratio.The camera trajectory specifies the output, with no temporal skips or zoom.Source: §3.1 · pp. 4–5

3.2 Data Collection: 360◦Test Videos and NFOV Training Videos

• The test collection contains 86 YouTube 360° videos totaling 7.3 hours, while the HumanCam collection contains 9,171 NFOV videos totaling 343 hours.Both collections use the same query terms; the HumanCam videos provide data for learning and evaluation.Source: §3.2 · p. 5, HumanCam NFOV videos · p. 5

3.3 AutoCam: Proposed Solution for the Pano2Vid Task

• AutoCam scores five-second fixed-direction NFOV glimpses for capture-worthiness, then selects a smooth path maximizing aggregate scores using dynamic programming.The classifier treats five-second HumanCam clips as positives and candidate glimpses from disjoint 360° videos as negatives; the authors note that this weak supervision may contain label noise.Source: Capture-worthiness of spatio-temporal… · pp. 6–7, Camera trajectory selection · pp. 7–8

3.4 Quantitative Evaluation of Pano2Vid Methods

• The evaluation measures resemblance to HumanCam videos and closeness to HumanEdit trajectories, using distinguishability, HumanCam-likeness, transferability, and trajectory or frame pooling.Transferability tests whether keyword-class classifiers trained on Auto or Human videos work on the other domain.Source: §3.4 · pp. 9–10, HumanEdit-based metrics · p. 10

3.5 HumanEdit Annotation Collection

• Annotators watched each full 360° video, then each provided two camera trajectories; 20 videos annotated by three people yielded 120 trajectories totaling 202 minutes.The interface let annotators direct a virtual NFOV camera by moving a cursor over an equirectangular panorama.Source: Annotation interface and collection… · p. 11
• HumanEdit trajectories differ by 59° on average, yet overlap exceeds 50% at every frame, indicating multiple natural paths and motivating metrics that account for them.The annotation set contains two trajectories from each annotator for each video.Source: HumanEdit consistency analysis · pp. 11–12, Table 1 · p. 12

4 Experiment

• The comparison evaluates Center + Smooth motion prior, Eye-level prior, Saliency trajectory, and AutoCam w/o stitching; each method generates K=20 outputs per 360° video, and results are averaged.Source: Baseline · p. 12

4.1 HumanCam-based Evaluation

• AutoCam performs consistently strongly across HumanCam-based and HumanEdit-based metrics, leading nearly all HumanCam metrics and all but frame-pooling overlap metrics.The authors describe this as evidence that AutoCam captures human-like virtual NFOV videos.Source: Table 3 · p. 13, §4.1 · p. 13, Table 4 · p. 14
• 35% higher Distinguishability and 23% higher HumanCam-Likeness than the best baseline; AutoCam’s Auto →Human Transferability advantage is 5%.AutoCam w/o stitching is second-best overall and outperforms the other three baselines.Source: Table 3 · p. 13, §4.1 · p. 13

4.2 HumanEdit-based Evaluation

• For each of two input panoramas, the examples show two generated trajectories, and the authors report multiple natural NFOV outputs per input.The reported results combine resemblance to generic YouTube NFOV videos with closeness to human-created edits.Source: Figure 5 · p. 13, §4.1 · p. 14, §4.2 · p. 14

4.1 HumanCam-based Evaluation

• Saliency performs better than Center and Eye-level but falls well short of AutoCam w/o stitching, making saliency a poor proxy for capture-worthiness.The Saliency baseline retains the smooth-motion selection pipeline while replacing capture-worthiness scores with saliency scores.Source: Baseline · p. 12, §4.1 · p. 13
• Transferability is asymmetric: AutoCam leads in Auto →Human, while AutoCam w/o stitching leads in Human →Auto; stitching may add less capture-worthy content between disjoint glimpses.The text suggests scene recognition may depend more on content selection than camera motion.Source: §4.1 · pp. 13–14

4.2 HumanEdit-based Evaluation

• On HumanEdit metrics, AutoCam w/o stitching suffers on cosine distance from incoherent motion, while Eye-level performs relatively better on frame-wise and overlap metrics.Eye-level even exceeds AutoCam on average per-frame overlap; Saliency scores poorly throughout.Source: Table 4 · p. 14, §4.2 · p. 14

5 Conclusion

• Pano2Vid is formulated as producing natural-looking NFOV video from dynamic panoramic 360° video.The paper presents the formulation as a new computer vision problem.Source: §5 · p. 14
• The work collects a dataset and performance metrics, then proposes AutoCam to learn camera trajectories from human-generated web video.The authors hope the work will provide a foundation for research requiring scene understanding and active decision making.Source: §5 · p. 14
• Future directions include supervised learning from HumanEdit data and incorporating task-specific features such as a human detector.Source: §5 · p. 14

A HumanCam Videos Collection

• The YouTube API search uses the “videoDuration” filter for short videos, defined as under four minutes.Source: A HumanCam Videos Collection · p. 17
• For each search term, the collection runs four queries with publish-time filters for 2016, 2015, 2014, and no date filter.The API returns about 500 results per query.Source: A HumanCam Videos Collection · p. 17
• Duplicate search results are removed using YouTube IDs.Source: A HumanCam Videos Collection · p. 17

B ST Glimpses Sampling

• ST glimpses are sampled densely in both space and time, balancing fine camera control with computation-resource constraints.Source: B ST Glimpses Sampling · p. 17
• Time and longitude φ are sampled uniformly.Source: B ST Glimpses Sampling · p. 17
• Latitude θ is sampled more densely around the equator because adjacent glimpses at the same polar angle overlap more at high latitude.Source: B ST Glimpses Sampling · p. 17

• The score distribution is almost binary, indicating a clear feature-space distinction between natural and unnatural content and supporting ST-glimpses as AutoCam’s basis.Source: C Capture-worthiness Analysis · p. 17
• Capture-worthy glimpses score ≥0.95 and non-capture-worthy glimpses score ≤0.05; the latitude and longitude plots show percentages of candidate glimpses.Source: C Capture-worthiness Analysis · p. 17
• Scores show no center preference but a weak eye-level preference, which the analysis links to camera recorders centering interesting content in some videos.Source: C Capture-worthiness Analysis · p. 17
• Video framing induces a universal eye-level preference, consistent with Eye-level performing better in Sec. 4.2, while off-level glimpses can still look natural.Source: C Capture-worthiness Analysis · p. 18

• The full equirectangular panorama stays visible so annotators can use content outside the camera FOV to direct it.Source: D HumanEdit Interface · p. 18
• Extending the panorama by 90° on each side places wraparound directions adjacent, avoiding the discontinuity at ±180°.Source: Figure 8 · p. 18, D HumanEdit Interface · p. 19
• Annotators steer the camera with the mouse during playback; its FOV is backprojected in cyan and mouse locations are recorded as HumanEdit.Source: D HumanEdit Interface · p. 19, Figure 9 · p. 19
• At ±180° edges, the mouse moves to a duplicate location while the duplicated camera center and FOV remain visible, simulating smooth motion.Source: Figure 10 · p. 19, D HumanEdit Interface · p. 20
• Before playback, annotators choose the display-center longitude to place content of interest in the middle of the screen.Source: D HumanEdit Interface · p. 20, Figure 11 · p. 20

• Each method generates K=20 trajectories to account for multiple valid outputs, which are evaluated jointly or independently depending on the metric.Source: E Generating Multiple Trajectories · pp. 20–21
• AutoCam computes an optimal trajectory for each last-frame glimpse location, then selects the K paths with the highest accumulated capture-worthiness scores.Intermediate-frame accumulated scores are reused, allowing all selected paths to be constructed in one dynamic-programming pass; Saliency uses the same procedure.Source: E Generating Multiple Trajectories · p. 20
• AutoCam w/o Stitching samples a glimpse independently at each time step, using a softmax of capture-worthiness scores to form its sampling distribution.Source: E Generating Multiple Trajectories · p. 21
• Center starts at θ=0, φ=0 and then follows random motion, with multiple trajectories generated through independent samples.Source: E Generating Multiple Trajectories · p. 21
• Eye-level holds θ at 0° and samples φ from 0° to 340° in 20° steps, yielding 18 trajectories rather than 20.Source: E Generating Multiple Trajectories · p. 21