Source-linked AI summary
Pano: Optimizing 360° Video Streaming with a Better Understanding of Quality Perception
Yu Guan, Chengyuan Zheng, Zongming Guo, Xinggong Zhang, Junchen Jiang
TL;DR
360° streaming consumes substantial bandwidth because existing approaches assume users perceive quality like non-360° video. Pano models viewpoint-dependent sensitivity, uses variable-sized tiling, and adapts quality dynamically. Compared with state-of-the-art techniques, it saves bandwidth or substantially raises perceived quality without using more bandwidth.
Problem
Existing 360° streaming solutions assume non-360° quality perception, limiting bandwidth-efficient delivery despite the high cost of streaming panoramic content.
Method
Pano combines a quality model for viewpoint speed, luminance change, and depth-of-field differences with variable-sized tiling and adaptive quality selection.
Results
25%-142% higher mean opinion score (MOS) is achieved over state-of-the-art solutions without using more bandwidth.
Takeaways & Limitations
Pano shows that exploiting 360°-specific quality perception can improve perceived quality or reduce bandwidth without requiring more bandwidth for higher quality.
Takeaways & Limitations
The evaluation varies viewpoint factors in controlled ways and assumes the effects of all three factors on JND are mutually independent.
Abstract
from arXiv · showhide
Streaming 360° videos requires more bandwidth than non-360° videos. This is because current solutions assume that users perceive the quality of 360° videos in the same way they perceive the quality of non-360° videos. This means the bandwidth demand must be proportional to the size of the user's field of view. However, we found several qualitydetermining factors unique to 360°videos, which can help reduce the bandwidth demand. They include the moving speed of a user's viewpoint (center of the user's field of view), the recent change of video luminance, and the difference in depth-of-fields of visual objects around the viewpoint. This paper presents Pano, a 360° video streaming system that leverages the 360° video-specific factors. We make three contributions. (1) We build a new quality model for 360° videos that captures the impact of the 360° video-specific factors. (2) Pano proposes a variable-sized tiling scheme in order to strike a balance between the perceived quality and video encoding efficiency. (3) Pano proposes a new qualityadaptation logic that maximizes 360° video user-perceived quality and is readily deployable. Our evaluation (based on user study and trace analysis) shows that compared with state-of-the-art techniques, Pano can save 41-46% bandwidth without any drop in the perceived quality, or it can raise the perceived quality (user rating) by 25%-142% without using more bandwidth.
1 INTRODUCTION
Pano addresses the bandwidth challenge of 360° video by modeling viewpoint-dependent quality perception and adapting encoding and delivery accordingly. Its evaluation reports higher perceived quality or lower bandwidth than existing approaches.
- 1 INTRODUCTION: 360° streaming can require 80× more bandwidth than a traditional full-HD screen-size video at the same perceived quality.The comparison uses roughly 5 Mbps for the traditional video and 400 Mbps for the panoramic sphere.
- 1 INTRODUCTION: Pano models sensitivity to quality distortion using viewpoint-moving speed, luminance change, and depth-of-field differences.The model is designed to estimate subjectively perceived quality more accurately than traditional metrics such as PSNR.
- 1 INTRODUCTION: Pano uses variable-sized tiles so regions with similar user sensitivity can be grouped without the inefficiency of uniformly coarse or fine tiling.This balances perceived quality against video encoding overhead.
- 1 INTRODUCTION: Pano dynamically predicts viewpoint movements and adapts quality while remaining robust to prediction errors and deployable in existing delivery infrastructure.Sensitivity estimation requires predicting ranges of viewpoint speed, luminance, and depth-of-field rather than exact viewpoint trajectories.
- 1 INTRODUCTION: 25%-142% higher mean opinion score (MOS) is achieved over a state-of-the-art solution without using more bandwidth.Pano was evaluated with user studies and trace-driven simulations across multiple video genres.
2 MOTIVATION
360° video streaming faces high bandwidth demands because viewport-driven systems assume non-360° quality perception. Pano instead exploits viewpoint-dependent sensitivity and spatially adapts quality to preserve perceived quality more efficiently.
- 2 MOTIVATION: The viewport-driven approach still needs at least twice the bandwidth of a screen-size video because a viewport is much wider than a laptop display.It also requires prefetching and fine-grained or trajectory-specific encoding, which can increase video size.
- 2.2 New quality-determining factors: Users are less sensitive to distortion when viewpoint movement is faster, enabling lower quality for static backgrounds while preserving tracked objects.This factor is especially relevant to sports videos with fast-moving objects.
- 2.2 New quality-determining factors: Luminance changes and depth-of-field differences can also permit quality reductions in regions without affecting perceived quality.Depth-of-field adaptation is particularly relevant when views switch between foreground objects and scenic backgrounds.
- 2.3 Potential gains: For 40% of the observed time, viewpoint speed exceeds 10 deg/sec, allowing users to tolerate 50% more distortion on background pixels than with computer-screen viewing.The measurements use real viewpoint traces from 18 360° videos watched by 48 users.
3 PANO OVERVIEW
Pano addresses three challenges in 360° video streaming by modeling viewpoint-dependent quality, using variable-sized tiles, and adapting quality robustly within existing delivery infrastructure.
- The system re-architects quality optimization and streaming components to exploit the distinct perception of 360° video quality.
- Challenge 1: Pano presents a quality metric that models user sensitivity as a function of viewpoint-moving speed, luminance change, and depth-of-field difference.
- Challenge 2: Pano uses variable-sized tiling to separate regions with different sensitivities while avoiding the encoding overhead of uniformly fine-grained tiles.
- Challenge 3: Pano adapts quality to tolerate bandwidth fluctuations and viewpoint-prediction errors while remaining deployable over the existing client-driven delivery infrastructure.
4 PANO: 360° VIDEO QUALITY MODEL
Pano extends perceived-quality modeling for 360° video by incorporating viewpoint-dependent sensitivity into PSPNR through a new 360JND formulation. User studies and validation indicate that the resulting metric captures quality perception more accurately than traditional alternatives.
- A general video quality framework: The quality framework uses PSPNR to filter out distortions below users’ just-noticeable differences rather than treating all pixel changes equally.
- Profiling JND of 360° videos: In 360° video, pixel-level JND depends on relative object speed, luminance relative to the previously focused viewpoint, depth-of-field difference, and content-dependent JND.
- Profiling JND of 360° videos: A user study with 20 participants measured JND across 43 short videos while varying quality distortion and the three viewpoint-related factors.
- 360JND: Pano defines 360JND as content-dependent JND multiplied by viewpoint-speed, luminance-change, and depth-of-field-difference factors.
- Validation of usefulness: 360JND-based PSPNR predicts mean opinion scores more accurately than traditional PSPNR and PSNR, indicating strong effects from the three 360°-specific factors.
5 PANO: VIDEO TILING
Pano replaces uniform tiling with variable-sized tiles formed by grouping fine-grained units according to their efficiency scores. The grouping seeks similar efficiency within each tile while reducing encoding overhead, but PSPNR estimation may be suboptimal for atypical viewing behavior.
- Tiling steps: Pano starts with a 12×24 grid of fine-grained square unit tiles and groups them into 30 variable-sized rectangular tiles.The unit tiles are one-second video clips covering square regions; the resulting coarse tiles are used for encoding.
- Tiling steps: Efficiency scores measure how quickly each unit tile’s perceived quality grows across quality levels.Pano uses these scores to assess which unit tiles should be grouped together.
- Tiling steps: Pano’s top-down grouping minimizes area-weighted variance in efficiency scores among unit tiles within each coarse-grained tile.The algorithm repeatedly splits rectangles horizontally or vertically until it obtains the target number of groups.
- PSPNR estimation: Pano estimates a lower bound on viewpoint-moving speed, which is often sufficient for accurate PSPNR estimation.Figure 10 compares the estimated lower bound with the actual viewpoint-moving speed.
- Limitations: Atypical viewing behavior can produce suboptimal quality, although the lowest perceived quality in the traces was at most 10% below the mean.This limitation concerns using average history to compute PSPNR.
6 PANO: QUALITY ADAPTATION
Pano adapts quality at both chunk and tile levels while using PSPNR to allocate quality under bitrate constraints. It makes this quality metric deployable in DASH by separating offline content processing from online viewpoint prediction.
- Quality adaptation: At the chunk level, MPC selects bitrate to meet a buffer target under predicted bandwidth, determining the total tile size available.The chunk bitrate constrains subsequent tile-level quality selection.
- Quality adaptation: Within each chunk, Pano chooses each tile’s quality level to maximize overall PSPNR while keeping total tile size below the chunk bitrate.The formulation weights tile quality by tile area.
- Quality adaptation: Pano avoids exhaustive enumeration of 5^N tile-quality assignments by pruning assignments that are strictly worse in both PSPNR and total size.The pruning rule compares pairs of tile assignments.
- Robustness: Lower bounds estimated from recent viewpoint history provide conservative inputs when exact future viewpoint speed, depth-of-field, and luminance are unpredictable.The lowest speed during the previous two seconds is given as an example estimator.
- DASH deployment: Pano decouples PSPNR computation into offline content preprocessing and online client-side viewpoint prediction for DASH deployment.The provider stores representative PSPNR values in a lookup table delivered through the manifest.
- DASH deployment: Pano compresses its PSPNR lookup table by replacing multidimensional factor combinations with products of their multipliers.The uncompressed table can reach 10 MB for a five-minute video.
7 IMPLEMENTATION
Pano’s implementation preprocesses video content and augments the manifest with tile metadata and PSPNR lookup information. The unchanged DASH server is paired with a modified client-side system.
- Video provider: The video provider extracts object trajectories, luminance, and depth-of-field features before augmenting the video manifest.These features support offline PSPNR calculation under historical viewpoint movements.
- Manifest metadata: Each tile’s manifest metadata includes its top-left coordinate, average luminance, average depth-of-field, object trajectories, and PSPNR lookup table.Tile coordinates are needed because Pano’s tiles may not align across chunks.
- System deployment: Pano does not require changes to the DASH video server, but it does require changes to the client-side system.The prototype was implemented using C++, C#, Python, and Matlab.
8 EVALUATION
Pano was evaluated through user studies and trace-driven simulations against viewport-driven and other baselines. It improved perceived quality, reduced bandwidth or buffering, remained effective under prediction errors, and added minimal system overhead.
- 8.2 End-to-end quality improvement: 25-142% higher MOS was achieved than the viewport-driven baseline with nearly the same bandwidth across seven 360° videos.Pano assigns higher quality to tracked moving objects and lower quality to static background regions that appear to move quickly.
- 8.2 End-to-end quality improvement: Across 18 videos, four genres, and two network traces, Pano achieved higher PSPNR, lower buffering ratio, or both than three baselines.The gains varied with viewpoint dynamics because more dynamic movements reduce sensitivity to quality distortion.
- 8.3 Robustness to prediction errors: Pano retained higher perceived quality than the baseline under viewpoint and throughput prediction errors, although quality degraded as throughput prediction error increased.The evaluation stress-tested noisy viewpoint trajectories and throughput predictions.
- 8.4 System overhead: Pano reduced client-side processing overhead and startup delay with minimal additional costs, while preprocessing time remained on par with the baseline.Its larger manifest adds overhead, but reduced first-chunk loading time offsets the additional startup delay.
- 8.5 Bandwidth savings: At PSPNR = 72, Pano saved 17% from JND awareness, 11% from 360JND factors, and 17% from variable-size tiling over the viewport-driven approach.With real throughput traces, Pano achieved the same PSPNR using 41-46% less bandwidth.
9 LIMITATIONS OF 360JND MODELING
The 360JND model has limitations because its factor variations may not match real-world viewing, and its factor interactions were only partially tested.
- 9 LIMITATIONS OF 360JND MODELING: The study varied viewpoint speed, depth-of-field, and luminance in specific ways that may not match users’ behavior in the wild.For example, simulated viewpoint movement was always horizontal and at a constant rate.
- 9 LIMITATIONS OF 360JND MODELING: Only two factors were tested at non-zero values, so Pano assumes the effects of all three factors on JND are mutually independent.The authors state that more complete and fine-grained profiling could improve the 360JND relationship.
10 RELATED WORK
Related work includes viewport tracking and tile-based bitrate adaptation for allocating quality across 360° video regions. Pano extends these approaches by predicting additional quality-determining factors and using them to determine tile importance.
- Viewport tracking: Viewport-driven streaming delivers the viewport in high quality while using lower quality elsewhere, with prediction schemes addressing viewpoint movement.Pano also predicts viewpoint-moving speed, luminance, and depth-of-field using ideas from prior viewport-prediction algorithms.
- Bitrate adaptation in 360° videos: 360° bitrate adaptation spatially allocates bitrate among tiles, whereas non-360° adaptation changes bitrate only between consecutive chunks.Pano follows tile-based adaptation but assigns tile importance using additional quality-determining factors.
11 CONCLUSION
Pano improves 360° video streaming by exploiting quality perceptions that differ from traditional video. It achieves higher perceived quality at the same bandwidth than state-of-the-art approaches.
- 25%-142% higher mean opinion score is achieved with the same bandwidth consumption as state-of-the-art techniques.
- Pano revisits perceived quality metrics, video encoding schemes, and quality adaptation logic for 360° video streaming.
- The system is motivated by the finding that users perceive 360° video quality differently from traditional non-360° video quality.
A APPENDIX
This appendix section introduces the methodology used to model just-noticeable differences in 360° video quality.
- The section presents the detailed methodology for modeling 360° video JND.
- The methodology focuses on modeling when viewers first notice quality distortion in 360° videos.
- The JND modeling methodology provides the basis for analyzing perceived quality distortion in 360° video.
A.1 Survey process
The survey process measures when participants first perceive controlled quality distortions in 360° videos. It uses repeated tests with manipulated videos and emulated viewpoint conditions.
- 20 participants watched 360° videos through an Oculus headset, but the limited participant pool may affect the results.
- JND is defined as the average distortion level first perceived by participants across repeated tests on 43 artificially generated videos.
- The study emulated relative viewpoint-moving speed, depth-of-field difference, and background luminance change while controlling video distortion.
- Controlled distortion was added to a 64×64-pixel grey foreground square by changing randomly selected pixel values.