Source-linked AI summary
Projection-Aware End-to-End Learned Video Compression for 360-Degree Video
Niloofar Maani
TL;DR
360-degree video must be projected to a 2D plane for conventional encoding, making projection choice central to compression performance. This thesis evaluates projection formats within an end-to-end neural compression pipeline and compares the results with traditional coding. Equirectangular-based formats perform best for the learning-based model, whereas cubemap-based formats perform better with HM-16.16, while projection-conversion differences affect the differentiable comparison.
Problem
360-degree video cannot be processed directly by standard codecs and requires projection to a 2D plane, but projection choice introduces efficiency and distortion trade-offs that require comparison.
Method
The study evaluates projected 360-degree sequences with the scale-space flow model and a three-layer pipeline that converts projections, compresses the coding projection, and converts it back.
Results
Equirectangular-based projections outperform cubemap-based projections for learning-based compression, whereas CMP outperforms ERP-based projections with traditional HM-16.16 coding.
Takeaways & Limitations
Projection efficiency is codec-dependent: single-face ERP formats support smoother motion vectors and higher compression efficiency for the learning-based model, while traditional codecs favor CMP-based layouts.
Takeaways & Limitations
Cubemap performance is constrained by uneven sample distribution across faces, and learning-based models can decline when trained on multi-face projection data without corresponding adaptation.
Abstract
from arXiv · showhide
360-degree video supports immersive applications such as virtual reality, autonomous driving, and education. Because spherical content cannot be processed directly by conventional video codecs, it must first be mapped to a two-dimensional projection. Projection choice affects spatial continuity, sampling uniformity, motion estimation, and compression efficiency. This thesis investigates how projection format influences end-to-end neural compression of 360-degree video. Seven formats supported by JVET 360Lib are evaluated using the scale-space flow model, JVET test sequences, and common test conditions. Each sequence is converted from its source equirectangular projection to a coding projection, compressed at multiple rate points, reconstructed, and converted back. Performance is assessed using PSNR, spherical PSNR, weighted spherical PSNR, and Bjøntegaard delta rate. A differentiable pipeline combining projection conversion, neural compression, and inverse projection is also compared with 360Lib. Results show that equirectangular and padded equirectangular projections provide the highest compression efficiency with the scale-space flow model, while cubemap-based and rhombic dodecahedron projections are less effective. This differs from the conventional HM-16.16 codec, for which cubemap-based formats, particularly equi-angular and adjusted cubemap projections, outperform equirectangular formats. Neural models based on optical flow benefit from the spatial continuity of single-face projections, whereas block-based hybrid codecs better accommodate multi-face layouts. These findings show that projection efficiency is codec-dependent and provide guidance for selecting projections for learning-based 360-degree video compression.
Declaration
The thesis declaration states that the work was written unaided, uses only listed sources, and has not previously been submitted elsewhere.
- The author confirms that the thesis was written unaided.
- The author states that all third-party material was acknowledged, whether quoted verbatim or used in substance.
- The declaration records the place, date, and author name as Niloofar Maani.
Kurzfassung
The section motivates end-to-end learned compression for projected 360-degree video and identifies the scale-space flow network as the evaluation model.
- 360-degree videos support applications including virtual reality, autonomous vehicles, and education, but their spherical form complicates standard processing.
- The study evaluates deep-learning-based end-to-end video compression networks in the presence of projection functions for 360-degree video.
- The scale-space flow network is used to assess the performance of the end-to-end compression approach.
Symbols and Notations
The notation list defines variables for optical flow, reconstructed and predicted frames, neural-network parameters, geometric transformations, and distortion evaluation.
- The notation includes optical-flow variables, quantized signals, reconstructed motion vectors, current frames, predicted frames, and reconstructed frames.
- Neural-compression notation includes residual data, convolution parameters and biases, downsampling factors, normalization terms, loss functions, and VAE parameters.
- Geometric and evaluation notation includes latitude, frame dimensions, tangent functions, reference and coding faces, rotations, distortion, the Lagrange multiplier, spherical factors, and radius.
Abbreviations and Acronyms
The abbreviations list expands terminology for projection formats, compression methods, coding standards, neural networks, and evaluation metrics.
- Projection-format abbreviations include ACP, CMP, EAC, ERP, and HEC.These denote adjusted cubemap, cubemap, equi-angular cubemap, equirectangular, and hybrid equi-angular cubemap projections.
- Compression and transform abbreviations include BD Rate, BDCT, DCT, EBCOT, and DVC.
- Neural-network and coding-standard abbreviations include CNN, DNN, FlowNet, GDN, and HEVC.
Introduction
360-degree video lets viewers look in all directions across a full spherical field of view, supporting immersive applications. Its high-resolution, high-frame-rate requirements create large files and processing challenges.
- 360-degree video lets users look around in all directions rather than seeing only the camera’s forward view.
- The format provides 360-degree horizontal and 180-degree vertical coverage for an immersive viewing experience.
- High resolution and frame rates produce large files, making 360-degree video difficult to store and transmit quickly.
Chapter 1 Introduction
The study examines how projection functions affect end-to-end neural compression of 360-degree video. It evaluates a scale-space flow pipeline and adds reprojection layers to connect coding projections with the source format.
- Motivation: Because conventional video standards process flat rectangles, spherical 360-degree content is projected onto a 2D plane before encoding.
- Motivation: Projection selection is important because different formats introduce distinct mapping distortions and affect compression efficiency.
- Method: The study evaluates projection-dependent compression efficiency using a scale-space flow network and JVET common test conditions.
- Method: The evaluation converts JVET sequences from their source projection to coding projections, compresses them, and converts reconstructions back.
- Method: A three-layer pipeline transforms the reference projection, compresses the coding projection, and converts the result back to the reference projection.
Learning-based Image and Video compression
This section reviews conventional and learning-based image and video compression, emphasizing end-to-end neural transforms, optical-flow motion modeling, and scale-space flow. It contrasts manually designed codec components with learned representations and motion compensation.
- Image compression: Image compression reduces bitrate by removing redundancy and irrelevant data while limiting quality loss.Transformation and prediction are central compression methodologies.
- Learning-based image compression: CNN-based image compression uses analysis and synthesis transforms as encoder and decoder, optimized end-to-end for rate-distortion performance.The analysis transform includes convolution, subsampling, and divisive normalization; quantization is approximated with uniform noise during training.
- Video compression: Traditional video codecs reduce temporal and spatial redundancy using block motion estimation, prediction, DCT, quantization, and entropy coding.Frames are classified as I-, P-, or B-frames, and motion compensation generates predicted frames from reference data.
- Learning-based video compression: DVC replaces conventional motion processing with CNN-based optical-flow estimation and learned motion-vector encoding and decoding.The estimated flow is discretized into motion data and reconstructed by an MV decoder before motion compensation.
- Learning-based video compression: DVC motion compensation warps a reconstructed previous frame using decoded flow to predict the current frame, then encodes the residual with a learned transform.A scale-space flow model was proposed to address the complexity of optical-flow compression architectures and training schemes.
360-degree Projection Formats
The thesis describes how 360-degree content is mapped between spherical coordinates and several 2D projection formats, each with distinct sampling, continuity, or distortion characteristics.
- 360-degree content must be projected onto a 2D plane before conventional coding, but projection can introduce redundancy or shape distortion.
- Each projection maps face, column, and row coordinates to 3D sphere coordinates through a defined forward and inverse conversion.
- ERP and PERP: ERP uses one face and maps its 2D coordinates through longitude and latitude to spherical coordinates.
- ERP and PERP: PERP duplicates samples across discontinuous boundaries to reduce seam artifacts in reconstructed viewports.
- Cubemap variants: CMP uses six square faces, but its uneven sampling concentrates samples near edges and reduces coding efficiency.
- Cubemap variants: EAC and ACP modify cubemap mapping to improve sampling uniformity, while HEC further addresses EAC's straight-line distortion.
- RSP: RSP uses two face rows with approximately six percent redundant corner samples that are deactivated for coding efficiency.
Material and methods
The evaluation converts JVET 360-degree sequences into candidate coding projections, compresses them with scale-space flow, converts reconstructions back to ERP, and evaluates spherical quality.
- Testing pipeline: The JVET procedure converts the source projection to a coding projection, compresses it, and converts the result back to the original projection for evaluation.
- Test data: The study evaluates 8K YUV(4:2:0) JVET sequences whose reference projection is ERP across the projection formats listed for comparison.
- Neural compression: Each coding projection is compressed with scale-space flow at eight quality levels, and bitrate is calculated for every level before inverse conversion to ERP.
- Projection conversion: 360Lib performs projection conversion face-by-face by mapping coding samples to 3D coordinates and then to reference-projection samples.
- Neural compression: The scale-space-flow model uses pre-trained CompressAI weights and expects RGB image tensors arranged as (N, C, H, W).
- Rate control: Rate points depend on λ, which trades bitrate against distortion; the model is fine-tuned for MSE and MS-SSIM across multiple bitrate levels.
- Quality evaluation: Evaluation uses PSNR and spherical quality metrics, including WS-PSNR, with projection-specific spherical factors accounting for geometric distortion.
4.5 Evaluation metrics
The evaluation uses spherical sampling and differentiable projection layers to measure 360-degree video quality and propagate compression losses through reprojection. The pipeline converts between ERP and coding projections before and after compression, with fixed projection parameters and nearest-neighbor mappings.
- Spherical PSNR: S-PSNR evaluates 360-degree video quality using uniformly distributed points on a sphere.Each spherical sample is mapped to corresponding positions in the reference and compressed frames before distortion is accumulated.
- Projection conversion: The projection pipeline contains fixed-parameter pre-processing, compression, and post-processing layers.Projection layers do not learn parameters; they convert ERP to a coding projection and back around the compression model.
- Projection conversion: The forward pass maps ERP points to CMP points, rounds non-integer coordinates, and selects corresponding pixel values.The mapping is not one-to-one, so some ERP points may remain unmapped and nearest-neighbor interpolation is used.
- Projection conversion: Backpropagation applies the chain rule through the projection layers to obtain input gradients and update the compression model.For the CMP layer, mapped ERP points receive gradient contributions while unmapped points have zero contribution.
Evaluation and Discussion
The experiments compare seven coding projections after converting ERP sequences, compressing them at multiple levels, and converting reconstructions back to ERP. With the scale-space flow model, ERP and Padded ERP outperform the other projections, whereas HM-16.16 favors cubemap-based formats.
- Projection analysis: ERP and Padded ERP show visible compression artifacts at the lowest compression level, reflecting quality loss in the projected outputs.The projections are compared at both the highest and lowest compression levels.
- Scale-space flow results: The cube-map formats CMP, ACP, EAC, and HEC have similar performance, while RSP and CMP show the lowest rate-distortion values.ERP and Padded ERP outperform the other projections, with EAC, HEC, and ACP between the leading and lowest-performing groups.
- Scale-space flow results: Positive BD-rate values indicate that ERP outperforms the corresponding cube-map or RSP projection.The BD-rate quantifies the size of the performance gap relative to ERP.
- Scale-space flow results: PERP is closest to ERP and outperforms ERP on all sequences except Harbor and PoleVault, while additional projection tests preserve the ERP/PERP advantage.The additional tests include AEP, GCMP, cube map 3x2 SEI, and cubemap 4x3.
- Comparison with HM-16.16: HM-16.16 reverses the projection ranking: CMP, ACP, RSP, and EAC outperform PERP, with EAC and ACP strongest and CMP weakest.This contrasts with the scale-space flow model, where ERP-based projections outperform CMP-based projections.
- Comparison of projection methods: The projection-layer method supports only CMP and produces lower end-to-end PSNR than the 360Lib method across the evaluated sequences.The two methods have similarly shaped rate-distortion curves, but 360Lib performs better.
Conclusion
This thesis evaluates end-to-end learned compression for projected 360-degree video and compares projection effects across learning-based and traditional codecs. It introduces a projection-aware evaluation pipeline and finds that codec architecture strongly influences which projections compress most effectively.
- Conclusion: The thesis evaluates deep-learning-based end-to-end compression networks for encoding and decoding projected 360-degree video frames.The evaluation uses the scale-space flow network and follows JVET common test conditions and procedures.
- Conclusion: A three-layer pipeline converts the reference projection to a coding projection, compresses it, and converts the reconstruction back.Projection conversion performs only input and output preprocessing, without learning parameters.
- Conclusion: Traditional codecs favored CMP-based projections, whereas the learning-based compression model favored ERP-based projections.The comparison included HM-16.16 and showed opposite projection preferences across codec types.
- Conclusion: Projection performance varied with codec architecture: hybrid codecs suited CMP's multiple faces, while scale-space flow performed better with single-face formats such as ERP.Single-face projections produced smoother motion vectors and higher compression efficiency for the learning-based model.
Chapter 6 Conclusion
The conclusion links projection performance to codec-specific motion and layout handling, while identifying projection conversion as an important source of differences between evaluation methods. It also points to further development of learning-based compression for varied projection formats.
- Chapter 6 Conclusion: Single-face projections such as ERP provide smoother motion vectors and higher compression efficiency for scale-space flow.This contrasts with the better suitability of CMP-based layouts for block-based hybrid codecs.
- Chapter 6 Conclusion: Training learning-based models on multiple perspectives may improve handling of CMP and other multi-angle projections.Such training could help models address changes in motion calculations and develop compression strategies for multi-angle formats.
- Chapter 6 Conclusion: The 360lib-based CMP projection method outperformed the model-based method in end-to-end PSNR.The discrepancy between ERP-to-CMP and CMP-to-ERP conversion methods was identified as the main reason for the performance gap.
- Chapter 6 Conclusion: The study provides insights into projection performance across traditional and learning-based codecs for future 360-degree video compression research.Its conclusions emphasize that projection effectiveness differs across compression approaches.