Source-linked AI summary
Viewport-Adaptive Navigable 360-Degree Video Delivery
Xavier Corbillon, Gwendal Simon, Alisa Devlic, Jacob Chakareski
TL;DR
360-degree streaming wastes bandwidth because HMDs display only a viewport while requiring head-movement responses within 10 ms. The paper proposes QER-based adaptive representations selected by bandwidth and viewing position, and finds cube-map layouts and short segments most effective for viewport quality and switching.
Problem
360-degree video delivers an omnidirectional stream although HMDs display only a viewport, while the 10 ms response requirement prevents server-side viewport updates after head movements.
Method
The system offers multiple QER-based representations, and the client selects one by expected throughput and proximity between its QEC and viewport center.
Results
The cube-map layout provides the highest viewport quality, while two-second segments support frequent view changes and deliver full quality three quarters of the time.
Takeaways & Limitations
Cube-map projection with two-second segments and six QERs offers the best reported performance under current encoding techniques.
Abstract
from arXiv · showhide
The delivery and display of 360-degree videos on Head-Mounted Displays (HMDs) presents many technical challenges. 360-degree videos are ultra high resolution spherical videos, which contain an omnidirectional view of the scene. However only a portion of this scene is displayed on the HMD. Moreover, HMD need to respond in 10 ms to head movements, which prevents the server to send only the displayed video part based on client feedback. To reduce the bandwidth waste, while still providing an immersive experience, a viewport-adaptive 360-degree video streaming system is proposed. The server prepares multiple video representations, which differ not only by their bit-rate, but also by the qualities of different scene regions. The client chooses a representation for the next segment such that its bit-rate fits the available throughput and a full quality region matches its viewing. We investigate the impact of various spherical-to-plane projections and quality arrangements on the video quality displayed to the user, showing that the cube map layout offers the best quality for the given bit-rate budget. An evaluation with a dataset of users navigating 360-degree videos demonstrates that segments need to be short enough to enable frequent view switches.
I. INTRODUCTION
360-degree video delivery wastes bandwidth because HMDs display only a viewport while requiring responses within 10 ms. The paper proposes viewport-adaptive representations that reduce distant-region quality while matching high quality to the user’s viewing position.
- HMDs display only a fraction of the omnidirectional video, so most delivered stream data are unused.
- The 10 ms response requirement prevents traditional server-adjusted delivery based on client head-movement feedback.
- A 120-degree viewport displayed at typically 4K requires a full 360-degree video of at least 12K resolution.
- The system offers representations with Quality Emphasized Regions, selecting one whose bit-rate fits bandwidth and whose Quality Emphasis Center is closest to the viewport center.
- The approach lowers delivered bit-rate while retaining full spherical video for immediate viewport extraction after head movements.
- The paper reports cube-map layout, two-second segments, and six QERs as the best-performing configuration under current encoding techniques.
II. BACKGROUND AND RELATED WORK
Spherical video must be projected onto a planar layout for conventional encoding, and different layouts introduce distinct sampling effects. The paper exploits layout geometry to encode a high-quality region around a chosen viewing position and evaluates viewport quality directly.
- Because current encoders operate on rectangular images, spherical video is projected onto a planar geometric layout before encoding.
- Equirectangular, cube-map, and rhombic-dodecahedron projections can generate viewports without information loss but may over-sample pixels.
- Pyramid projection under-samples spherical pixels, causing distortion and information loss in extracted viewports.
- The proposed encoding uses each layout’s central point to keep the front face at full quality while reducing quality on other faces.
- The evaluation measures QoE from several extracted viewports rather than from the full projected video.
B. Personalized Viewport-Only Streaming
Streaming only the current viewport could reduce resource waste but cannot satisfy rapid navigation because a head movement would require newly received content. The proposed alternative delivers a complete, QER-dependent video and performs representation selection and viewport extraction at the client.
- Viewport-only streaming cannot meet the 10 ms latency requirement because head movements require server notification and newly adjusted viewport delivery.
- Viewport-only delivery also requires the server to extract a video portion separately for each client connection.
- The proposed system always delivers the full video through versions associated with different QERs and QECs.
- The client selects the representation and extracts the viewport, shifting storage requirements to the server while keeping processing on the client.
C. Tiling for Adaptive Video Streaming
Tile-based adaptive streaming preserves navigability and bandwidth savings but introduces reconstruction, encoding, server-management, and client-management costs. The QER-based approach uses fewer representations and simpler selection without reconstructing tiles before viewport extraction.
- Tile-based delivery requires reconstructing independent tiles before viewport extraction, consuming energy and time for each video frame.
- Independent tiles reduce encoding efficiency and increase server file and manifest-management overhead.
- An 8 × 8 tiling scheme with six quality levels produces 384 independent files for each video segment.
- QER-based representations simplify server management and client selection and avoid video reconstruction before viewport extraction.
D. QER-Based Streaming
The paper positions QER-based streaming alongside related navigable-video systems and identifies its broader study scope. Its evaluation extends across geometric layouts, quality arrangements, segment duration, QER count, and MPEG DASH integration.
- Facebook’s pyramid-based delivery system is similar because its front face has higher quality and users periodically select representations by viewport center.
- Prior QER coding for regular video places emphasis near likely gaze areas but does not generate multiple representations with different QERs.
- The paper studies multiple QER-based representations while additionally varying geometric-layout quality and evaluating layouts, segment duration, QER count, and MPEG DASH integration.
III. SYSTEM ARCHITECTURE
The system encodes 360-degree video into representations combining different QER locations and bit-rates, then lets the client select segments using viewport and throughput. This extends DASH adaptation with QER-aware selection while preserving the full sphere for viewport extraction.
- Server: The server transforms equirectangular video into a geometric layout and creates representations spanning different QERs and bit-rates.Each representation is segmented for switching; quality is highest around its QEC and lower elsewhere.
- Client: The client requests each new segment from the representation matching both the new viewport center and available throughput.
- Adaptation algorithm: The adaptation algorithm first selects the QER nearest the viewport center, then chooses its bit-rate representation according to expected throughput.The selected representation supplies the spherical video from which the client extracts the viewport.
- Adaptation algorithm: QER-aware adaptation must keep viewport centers close to the chosen QEC during the segment, in addition to matching requested bit-rate to throughput.
- Video segment length: Segment duration trades easier management and encoding against faster resynchronization to head movement and bandwidth changes.Typical segment lengths range from 1 s to 10 s.
- MPD integration: Each representation’s MPD metadata includes its QEC spherical coordinates alongside standard DASH parameters.The coordinates specify the QEC’s two spherical angles.
IV. SYSTEM SETTINGS
The study uses a software tool and real-user head-movement data to examine system parameters. The tool supports projection, face-level quality adjustment, and viewport extraction across layouts with differing face qualities.
- The experiments vary global parameters such as QER count, representation count, segment length, and geometric layout, plus local quality and bit-rate settings.
- The authors use a software tool and a real VR-system dataset to study segment length, QER count, and related configuration questions.
- Dataset: The dataset contains eleven ten-second omnidirectional videos and head movements from eleven users watching freely with an Oculus Rift DK2.The authors characterize these movements as a challenging wide-head-movement configuration.
- Software: The software projects between spherical video and four geometric layouts, rotating content so the QEC remains at a fixed 2D position.
- Software: The tool assigns each geometric face a resolution and target encoding bit-rate to control its video quality.
- Software: The tool decodes, rescales, and projects faces to extract viewports, including viewports overlapping faces with different resolutions and bit-rate targets.
A. Geometric Layout
The study compares quality-differentiated spherical projections against a uniform equirectangular baseline using viewport quality and sensitivity to QEC distance. At a 6 Mbps budget, the cube map provides the strongest quality across tested distances.
- The evaluation measures best viewport quality at matching QEC and viewport centers, and sensitivity as their distance increases.MS-SSIM and PSNR compare extracted viewports with original full-quality viewports.
- The experiment compares viewport quality from quality-differentiated equirectangular, cube-map, pyramid, and dodecahedron layouts against full-quality and uniform baselines.The uniform equirectangular video provides a distance-independent reference at the target bit-rate.
- The cube-map arrangement uses full quality on one face and 25 % of that target on the other faces.
- At d = 0, cube-map viewport quality exceeds 0.98, corresponding to imperceptible distortion relative to the full-quality video.
- The cube map has the highest quality across distances, while pyramid projection is especially sensitive to head movements.
- Cube-map quality exceeds the uniform equirectangular baseline for viewport centers up to 2 distance units from the QEC.
B. Segment Length
Short segments are central to viewport-adaptive streaming because they allow faster resynchronization between head movement and quality-emphasized regions. The evaluation indicates that segments typically shorter than 3 s, especially 2 s, balance quality and responsiveness.
- Segment length should keep the viewport center rarely more than 1.5 distance units from the quality-emphasized center.
- The movement analysis measures orthodromic distance from the segment's initial head position across the following segment duration.
- 5 s segments leave users watching more than 1.3 distance units from the initial head position for about half their time, degrading video quality.
- 2 s segments are a good trade-off: 92 % of users stayed within 2 distance units, while full quality was available three quarters of the time.
C. Number of QERs
Increasing the number of quality-emphasized regions improves viewport coverage but raises server storage and management costs. In the evaluated configuration, 5–7 QERs provide the best trade-off, with limited gains beyond that range.
- More QERs improve viewport quality through better matching between the quality-emphasized center and viewport center, but increase storage and management requirements.
- The best number of QERs in this configuration is between 5 and 7.
- Higher QER counts do not provide gains significant enough to justify their storage requirements, particularly 30 QERs.
- Multiple QERs yield higher quality gains for short segments because they improve resynchronization between QERs and viewport centers.
V. CONCLUSION
The paper introduces viewport-adaptive streaming for navigable 360-degree video and studies its main system settings. It identifies cube-map projection with 2 s segments and six QERs as the best-performing configuration under current encoding techniques.
- The system combines viewport-adaptive streaming with cube-map projection, 2 s segments, and six QERs for its best current performance.
- The conclusion identifies head-movement prediction, quality-differentiated encoding, and live or interactively generated VR streaming as open research directions.