Source-linked AI summary
Adaptive 360 VR Video Streaming: Divide and Conquer!
Mohammad Hosseini, Viswanathan Swaminathan
TL;DR
360 VR videos require substantial bandwidth, making acceptable-quality streaming difficult for bandwidth-constrained devices. The paper proposes dynamic FoV-aware tile adaptation using MPEG-DASH SRD and a hexaface sphere, reporting up to 72% bandwidth savings with minor negative quality impacts.
Problem
360 VR video streaming faces high bitrate requirements, while the semantic link among users’ FoV, spatial partitioning, and stream prioritization remains insufficiently developed for bandwidth management.
Method
The system spatially tiles 360 VR videos and their 3D geometry, describes tile relationships with MPEG-DASH SRD, and prioritizes viewport tiles using dynamic view-aware adaptation and a hexaface sphere mesh.
Results
Up to 72% bandwidth savings were achieved for 360 VR video streaming without much noticeable quality impact compared with no adaptation.
Takeaways & Limitations
The proposed adaptations provide bandwidth-efficient 360 VR streaming by assigning higher quality to the user’s immediate viewport and lower resolutions elsewhere.
Abstract
from arXiv · showhide
While traditional multimedia applications such as games and videos are still popular, there has been a significant interest in the recent years towards new 3D media such as 3D immersion and Virtual Reality (VR) applications, especially 360 VR videos. 360 VR video is an immersive spherical video where the user can look around during playback. Unfortunately, 360 VR videos are extremely bandwidth intensive, and therefore are difficult to stream at acceptable quality levels. In this paper, we propose an adaptive bandwidth-efficient 360 VR video streaming system using a divide and conquer approach. In our approach, we propose a dynamic view-aware adaptation technique to tackle the huge streaming bandwidth demands of 360 VR videos. We spatially divide the videos into multiple tiles while encoding and packaging, use MPEG-DASH SRD to describe the spatial relationship of tiles in the 360-degree space, and prioritize the tiles in the Field of View (FoV). In order to describe such tiled representations, we extend MPEG-DASH SRD to the 3D space of 360 VR videos. We spatially partition the underlying 3D mesh, and construct an efficient 3D geometry mesh called hexaface sphere to optimally represent a tiled 360 VR video in the 3D space. Our initial evaluation results report up to 72% bandwidth savings on 360 VR video streaming with minor negative quality impacts compared to the baseline scenario when no adaptations is applied.
I. INTRODUCTION
360 VR video provides an immersive spherical viewing experience but is difficult to stream at high resolution under bandwidth constraints. The paper proposes view-aware, tiled streaming that connects the viewport to spatial partitioning and tile prioritization.
- Motivation: 360 VR video maps spherical content onto 3D geometry, allowing users to look around during playback.The user views the content through a head-mounted display.
- Motivation: High-resolution 360 VR video is extremely bandwidth intensive and difficult to deliver to bandwidth-constrained devices.The challenge is especially significant for wireless VR head-mounted displays and resolutions such as 4K, 8K, and higher.
- Research gap: The paper addresses an underdeveloped link among Field of View, spatial partitioning, and stream prioritization for bandwidth management.The proposed dynamic adaptations use this semantic link for high-performance 360 VR streaming.
- Approach: The system divides videos into tiles, describes their 360-degree spatial relationships with MPEG-DASH SRD, and prioritizes viewport tiles.It also partitions the underlying 3D mesh and constructs a hexaface sphere for tiled 3D representation.
II. BACKGROUND AND RELATED WORK
The paper situates its approach within prior work on multimedia streaming concepts and categories.
- The related-work section introduces concepts and categories relevant to the proposed approach.
A. Dynamic Adaptive Streaming
Adaptive streaming selects video quality according to constrained client resources, while MPEG-DASH SRD represents spatial relationships among tiled multimedia content. These ideas support spatially adaptive streaming for immersive video.
- Dynamic Adaptive Streaming: Adaptive streaming lets clients choose video segments with suitable quality based on available constrained resources such as bandwidth.Dynamic Adaptive Streaming over HTTP is identified as MPEG-DASH, an ISO standard.
- Dynamic Adaptive Streaming: MPEG-DASH SRD describes spatial relationships among spatially partitioned multimedia content.It combines spatially related videos into tiled representations while retaining compatibility with regular adaptation sets.
- Dynamic Adaptive Streaming: A 90-degree spherical viewport can be mapped onto the corresponding region of a raw 360-degree video frame.
B. Prioritized Multimedia Streaming
Prioritized multimedia streaming assigns different quality levels to content regions according to their importance, especially the user’s viewport. The paper builds on this principle for adaptive 360 VR streaming.
- Prioritized Multimedia Streaming: Prior work assigns higher quality to 3D content parts within the user’s viewport to support resource-limited devices.
- Prioritized Multimedia Streaming: The paper builds on related prioritization concepts to implement an adaptive, prioritized, view-aware approach for reducing 360 VR video bandwidth requirements.
III. METHODOLOGY
The methodology maps 360 VR video onto a 3D sphere, partitions the geometry into tiles, and uses viewport-aware quality allocation to reduce redundant streaming. It introduces the hexaface sphere as a configurable geometry for tiled 360 VR representation.
- 360 VR video is mapped as a 3D texture onto a sphere, while users view only a confined portion corresponding to their viewport.
- The system assigns higher quality to viewport tiles and lower quality to tiles outside the immediate viewport.This targets the high bandwidth demands of 360 VR streaming while supporting high-resolution viewport content.
- A procedural sphere mesh uses slices and stacks to control spherical smoothness, triangle count, and rendering speed.Algorithm 1 generates spatial points for the mesh from the stack and slice parameters.
- The sphere is partitioned into top cap, middle body, and bottom cap, with the middle body further split into meshes covering α degrees of the 360-degree screen.The number of partitions can depend on the VR HMD’s horizontal FoV.
- The resulting combination of six 3D meshes is called a hexaface sphere 3D mesh.The α = π/2 and β = π/4 settings reflect device defaults and performance-quality trade-offs, but the approach supports more meshes and other projections.
B. Viewport Tracking
Viewport tracking converts VR orientation into segment-specific quaternion confines, identifies mesh segments intersecting the viewport, and adapts tile bitrate dynamically. Tiles inside the FoV receive higher bitrate content while other tiles receive lower bitrate content.
- VR device orientation is converted from Euler angles to a unit quaternion and then to a normalized direction vector.
- Segment-specific quaternion confines are computed to determine which hexaface sphere segments intersect the user’s viewport.
- Viewport tracking runs at every frame and drives view-aware adaptation of the corresponding video tiles.The system dynamically delivers higher bitrate content to tiles within the user’s FoV and lower bitrate content elsewhere.
IV. EVALUATION
The evaluation uses a Samsung Gear VR prototype and compares view-aware tiled streaming with hierarchical representations against tiled streaming without adaptation. The adaptations reduce bandwidth while preserving viewport quality with minor visible impacts.
- Platform and setup: The evaluation targets a Samsung Gear VR HMD with a Galaxy S7, 2560x1440 total resolution, 60 FPS maximum frame rate, and 96-degree horizontal FoV.The prototype uses MPEG-DASH SRD with Oculus Mobile SDK 1.0.3 and Android SDK API 24.
- Encoding and packaging: The system generates six spatially cropped tiles for the hexaface sphere and encodes video segments into four H.264 AVC representations.The representations use hierarchical resolution degradation with polynomially decreasing resolutions.
- Evaluation procedure: Experiments assign maximum resolution to tiles in the immediate FoV and lower resolutions elsewhere, measuring normalized average bitrate against tiled streaming with no adaptation.Each trial lasts 30 seconds, with each tile inside the user’s viewport for 5 seconds.
- Bandwidth results: 72% bandwidth savings are achieved compared with tiled streaming without viewport-aware adaptation.The baseline streams all tiles using the highest representation, REP1.
- Visual quality: The Waldo visual comparison uses REP4 at 240x480 for peripheral tiles while retaining the highest representation for the main front tile.The reported quality impacts are minor and sometimes imperceptible.
V. CONCLUSION AND FUTURE WORK
The paper concludes that FoV-aware, bandwidth-efficient adaptations address the high bandwidth demands of 360 VR streaming through spatial partitioning and dynamic viewport adaptation. Initial experiments report substantial savings with limited visible quality impact, while future work will examine more 3D segments and trade-offs.
- Conclusion: The proposed system combines spatially partitioned 360 VR video and 3D geometry with dynamic viewport adaptation.The approach uses the semantic link between MPEG-DASH SRD and the user’s viewport.
- Conclusion: 72% bandwidth savings are reported for 360 VR streaming without much noticeable quality impact.
- Conclusion: The adaptive rate allocation algorithm selects tile bitrates given the available bandwidth for tiled streaming.
- Future work: Future work will extend the hexaface sphere to more 3D segments and measure bandwidth-saving versus performance trade-offs.
VI. APPENDIX I: RATE ALLOCATION PROBLEM
The rate allocation problem is formulated as selecting bitrate representations for all necessary tiles under a bandwidth budget. Viewport priorities determine quality contributions, with highest priority assigned inside the immediate viewport and lowest priority outside it.
- Problem formulation: The system models rate selection as a multiple-choice knapsack problem whose groups contain alternative bitrate versions of each tile.Unlike binary knapsack selection, the proposed formulation retains all necessary tiles to avoid black views when the user changes orientation.
- Problem formulation: Each tile receives a priority coefficient, and its quality contribution is modeled as q_τh = p_τh × s_τh under an available bandwidth budget W.The model includes n tiles in the 360-degree VR environment.
- Tile prioritization: Tiles inside the immediate viewport receive highest priority C1, while tiles outside the viewport receive lowest priority C3.The pilot study uses three priority classes but the approach can support any number of classes.
- Algorithm output: The algorithm outputs adapted tiles X, with each output tile corresponding to an original tile and retaining its priority coefficient.
- Representations: Each tile has L+1 manifest representations, with RL denoting the lowest-bitrate representation tolerated by users.
A. Heuristic Algorithm
The heuristic initializes every tile at its minimum representation, then distributes remaining bandwidth to prioritized tiles while preserving coverage. When the viewport intersects multiple tiles, visible tiles are grouped to receive the same bitrate.
- Initialization: The heuristic first assigns all tiles their minimum bitrate, computes Wmin, and uses the remaining budget W0 = W − Wmin for upgrades.
- Greedy allocation: Tiles are sorted from highest to lowest global priority, and each is upgraded to its highest representation when the remaining budget allows.
- Multi-tile adaptation: When a viewport intersects multiple tiles, the heuristic packs all visible tiles together and assigns them the same bitrate.This avoids edging problems caused by quality variation among visible tiles.
- Greedy allocation: When the next tile cannot receive its highest bitrate, the algorithm selects the highest feasible representation within the remaining budget and continues with later tiles.
- Complexity and approximation: The implementation runs in O(nlogn) time and O(n) space, produces solutions close to optimal, and has approximation error tied to the first tile that cannot be received at highest bitrate.