PaperlayerView exact revision on arXiv

Source-linked AI summary

Making 360$^{\circ}$ Video Watchable in 2D: Learning Videography for Click Free Viewing

Yu-Chuan Su, Kristen Grauman

arXiv:1703.00495v2cs.CV

TL;DR

Viewers of 360° video must steer a virtual camera without seeing beyond its current field of view, making it difficult to choose what to watch. The paper extends Pano2Vid with zoom and diverse outputs and introduces efficient search; experiments report better results and lower runtime than existing methods and strong baselines.

  • Problem

    Viewers must choose where and what to look at through a virtual camera without information about content beyond its current field of view.

    Source: §1 · p. 2

  • Method

    The method generalizes Pano2Vid to control field of view, uses coarse-to-fine trajectory search, and generates diverse plausible NFOV outputs.

    Source: §1 · p. 2

  • Results

    Autozoom improves results by up to 43.4%, and the method reduces runtime by more than 84% compared with the existing solution and strong baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across a battery of metrics, including comparisons with human editors, the algorithm outperforms the state of the art on unrestricted 360° video.

    Source: §6 · p. 8

    The learned capture-worthiness does not capture context-induced preferences, and the smoothness constraint can be too strong in scenes with rapidly changing content.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

360$^{\circ}$ video requires human viewers to actively control "where" to look while watching the video. Although it provides a more immersive experience of the visual content, it also introduces additional burden for viewers; awkward interfaces to navigate the video lead to suboptimal viewing experiences. Virtual cinematography is an appealing direction to remedy these problems, but conventional methods are limited to virtual environments or rely on hand-crafted heuristics. We propose a new algorithm for virtual cinematography that automatically controls a virtual camera within a 360$^{\circ}$ video. Compared to the state of the art, our algorithm allows more general camera control, avoids redundant outputs, and extracts its output videos substantially more efficiently. Experimental results on over 7 hours of real "in the wild" video show that our generalized camera control is crucial for viewing 360$^{\circ}$ video, while the proposed efficient algorithm is essential for making the generalized control computationally tractable.

1. Introduction

360° capture lets videographers record the whole visual world and viewers explore it, but watching through a virtual camera makes viewers responsible for choosing where and what to see. The paper extends Pano2Vid with zoom, efficient trajectory search, and diverse outputs; its experiments report improved results and lower runtime.

1. Introduction

  • 360° capture frees videographers from choosing a single direction while recording and lets viewers explore content according to their interests.The paper contrasts this freedom with traditional cameras’ restricted field of view and describes 360° cameras as capturing the visual world from their optical center.Source: §1 · p. 1
  • Viewers must steer a virtual camera to choose what they see, yet cannot see beyond its current field of view and may need repeated viewings to find a suitable path.Displaying the whole video with equirectangular projection is also difficult because the format is unfamiliar and distorted.Source: §1 · pp. 1–2
  • The method generalizes Pano2Vid to control both spatial selection and field of view, uses coarse-to-fine search, and generates diverse outputs to address multimodal solutions.Source: §1 · p. 2
  • Autozoom improves results by up to 43.4%, while the proposed method reduces runtime by more than 84%.Experiments use more than 7 hours of real-world 360° video and 12 hours of manually edited 360° data.Source: §1 · p. 2

2. Related Work

Related work spans saliency, retargeting, virtual cinematography, summarization, and diverse-solution search. Pano2Vid differs in its spatial framing of unrestricted 360° input and its data-driven control of plausible camera views.

Video saliency

  • Unlike saliency, which predicts eye-catching 2D locations from local image content, Pano2Vid predicts camera directions in spherical coordinates using content and composition across the full field of view.Source: §2 · p. 2

Video retargeting

  • Unlike retargeting, which crops an already edited video to preserve its information, Pano2Vid takes unedited 360° input and generates multiple spatially distinct outputs.Pano2Vid also reduces spatial extent more severely than typical web-video-to-mobile retargeting.Source: §2 · p. 2

Virtual cinematography

  • Prior virtual cinematography often targets virtual environments or specialized domains and uses hand-crafted rules; this work learns from web videos and adds zoom and diverse hypotheses to AUTOCAM.Its input is unrestricted real 360° video.Source: §2 · p. 2

Video summarization

  • Video summarization selects content temporally, whereas Pano2Vid selects it spatially and produces continuous camera-like videos rather than keyframes or disjoint clips.Source: §2 · pp. 2–3

Diverse solution search

  • Unlike diversity methods that penalize similarity between successive approximate or probabilistic solutions, this approach uses exact dynamic-programming search and guarantees minimum distance between solutions.Source: §2 · p. 3

3. Preliminaries: AUTOCAM

AUTOCAM samples fixed-direction, five-second NFOV clips, scores their likelihood of appearing in human-captured video, and selects a smooth high-scoring camera trajectory. Its search uses dynamic programming to produce multiple outputs.

3. Preliminaries: AUTOCAM

  • AUTOCAM samples five-second ST-glimpses at 65.5° FOV across 18 azimuthal and 11 polar angles every five seconds.Each candidate is identified by its camera direction and time in the video.Source: §3 · p. 3
  • A classifier trained to distinguish human-captured NFOV clips from ST-glimpses supplies capture-worthiness scores for candidate clips.The method assumes human-captured NFOV clips are mostly capture-worthy and most ST-glimpses are not; it uses YouTube NFOV videos and C3D features.Source: §3 · p. 3
  • AUTOCAM finds a trajectory maximizing aggregate capture-worthiness while enforcing smooth motion, then uses dynamic programming to generate the top K NFOV outputs.In practice, each new glimpse must lie within the previous glimpse’s eight-neighborhood; the search considers 198 possible locations.Source: §3 · p. 3

4. Approach

The approach generalizes Pano2Vid to variable-FOV camera control, reduces trajectory-search cost, and generates diverse output trajectories. It proceeds from zoom-capable camera control to coarse-to-fine search and iterative diversity search.

4. Approach

  • The approach first extends Pano2Vid with variable-FOV control and efficient optimization, then expands the best single output into diverse plausible trajectories.Multiple outputs address the existence of different valid presentations of the same video and can provide editors with different initializations.Source: §4 · p. 3, §4.3 · p. 5

4.1. Zoom Lens Pano2Vid

  • Zoom Lens Pano2Vid samples ST-glimpses at different focal lengths, then searches trajectories across direction and focal-length grids with gradual zoom changes.The method uses three focal lengths, yielding FOVs of 104.3°, 65.5°, and 46.4°; gradual focal-length changes encode a prior about human videographers.Source: §4.1 · pp. 3–4

4.2. Coarse-to-Fine Camera Trajectory Search

  • Coarse-to-fine search builds a trajectory from coarsely sampled glimpses, then refines it around that path using dense samples, reducing the number of scores computed.The coarse trajectory uses a single focal length, with zoom enabled during refinement; this preserves effective candidate coverage without processing every candidate glimpse.Source: §4.2 · p. 4, Figure 4 · p. 4

4.3. Diverse Camera Trajectory Search

  • Iterative diversity search samples time windows, excludes previously selected glimpses within each window, and chooses the best resulting trajectory for each iteration.The window length sets the minimum distance between outputs; sampling multiple windows helps avoid excluding critical glimpses from the search space.Source: §4.3 · p. 5, Figure 5 · p. 5

5. Experiments

Experiments evaluate Pano2Vid on real 360° videos using human-edited trajectories and human-captured videos as references. The full method improves output quality over baselines, while coarse-to-fine search reduces computational cost and diverse search produces more distinct trajectories.

5.1. Dataset

  • The evaluation uses 86 YouTube 360° videos totaling 7.3 hours and 480 HumanEdit trajectories from 40 videos, annotated by three editors each.HumanCam data comprises 343 hours of NFOV YouTube videos.Source: §5.1 · p. 6

5.3. Evaluation Metrics

  • Evaluation compares algorithm outputs with human-captured videos and human-edited trajectories, and also measures processing time and trajectory diversity.The HumanEdit-based metric computes camera-FOV overlap using trajectory-level or frame-level pooling.Source: §5.3 · pp. 6–7

5.4. Output Quality

  • OURS improves Distinguishability by 43.4%, HumanCam-Likeness by 25.5% on average, and Trajectory Overlap by 23% over AUTOCAM.The authors attribute the gains to the proposed zoom lens.Source: §5.4 · p. 7, Table 1 · p. 7
  • OURS often selects a 104.3° FOV: 76% of its selected ST-glimpses use that FOV, compared with 55% of HumanEdit data.The authors suggest larger FOV may be preferable because objects of interest are often closer to the camera.Source: §5.4 · pp. 7–8

5.2. Baselines

  • CENTER and EYE-LEVEL perform poorly, while SALIENCY underperforms OURS on all metrics except Human →Auto transferability.The authors characterize gaze-attracting content as a poor proxy for Pano2Vid.Source: §5.4 · p. 8

5.5. Computational Cost

  • OURS W/ FAST cuts OURS’s computational cost by 84%, performs similarly in Trajectory Overlap, and is 6% worse in Distinguishability.It takes less than 50% of AUTOCAM’s computation while outperforming it in all reported metrics.Source: §5.5 · p. 8, Figure 8 · p. 8

5.6. Output Diversity

  • The diverse trajectory search generates 2–3 times more distinct outputs than the ablated non-diverse variant.SALIENCY produces fewer than two distinct trajectories on average among its top 20 outputs.Source: §5.6 · p. 8, Figure 9 · p. 8

6. Conclusion

The paper introduces virtual cinematography for 360° video, with dynamic FOV control, computationally efficient trajectory search, and diverse outputs. Across its evaluations, the algorithm outperforms the state of the art, bringing practical virtual videography closer.

6. Conclusion

  • The system controls a virtual camera in 360° video to produce human-captured-looking videos for passive viewers.Source: §6 · p. 8
  • The method generalizes Pano2Vid to dynamic FOV, uses coarse-to-fine optimization, and encourages diverse output trajectories.Source: §6 · p. 8
  • Across a battery of metrics, including comparisons with human editors, the algorithm outperforms the state of the art.Source: §6 · p. 8

Appendix

The appendix lists additional metric details, computational-cost results, and information about the annotation interface and its demonstration.

A. HumanEdit-Based Metric

B. Computational Cost Results

  • Appendix B presents complete computational-cost experiment results.Source: Appendix · p. 10

C. HumanEdit Interface

  • Appendix C introduces the annotation interface, with a project webpage offering a demonstration and example output videos.Source: Appendix · p. 10

A. HumanEdit-Based Metric

The HumanEdit-based metric pools framewise overlap in two ways: across whole trajectories or separately at each frame.

A. HumanEdit-Based Metric

B. Computational Cost Results

OURS W/ FAST significantly outperforms AUTOCAM across all reported metrics, while showing a narrower trade-off against OURS across quality measures.

B. Computational Cost Results

C. HumanEdit Interface

The HumanEdit interface presents panoramic video while allowing editors to control a virtual NFOV camera and its focal length as they create trajectories.

C. HumanEdit Interface

  • The interface displays the video in equirectangular projection and extends the panoramic strip by 90° on both sides to mitigate edge discontinuities.Source: C. HumanEdit Interface · p. 10, Figure 11 · p. 11
  • Editors direct the virtual NFOV camera with the cursor, see its frame boundaries backprojected onto the strip, and can change focal length.Source: C. HumanEdit Interface · p. 10
  • Editors first watch the full video, then annotate four trajectories in passes with the strip panned by 0°, 180°, 0°, and 180°.Source: C. HumanEdit Interface · p. 10
  • For the first two videos annotated by each editor, rendered outputs are shown after the first two trajectories to illustrate the resulting video.Source: C. HumanEdit Interface · p. 10
  • The interface controls and example output videos are demonstrated on the project webpage.Source: C. HumanEdit Interface · p. 10
Loading 1703.00495v2…