Projection-Aware End-to-End Learned Video Compression for 360-Degree Video

• The author confirms that the thesis was written unaided, uses no unlisted sources, acknowledges third-party content, and was not previously submitted for examination.Source: Declaration · p. 5
• The declaration is signed by Niloofar Maani and includes a place-and-date line.Source: Declaration · p. 5

• 360-Grad-Videos müssen wegen ihrer Kugelform vor der Verarbeitung mit Standard-Videocodierungstechniken in 2D-Projektionen überführt werden.Die Arbeit vergleicht daher verschiedene JVET-Projektionstechniken hinsichtlich ihrer Kompressionseffizienz.Source: Kurzfassung · p. 9
• Die Studie bewertet die Leistung end-to-end optimierter Deep-Learning-Videokompression mit dem Scale-Space-Flow-Netzwerk bei 360-Grad-Projektionen.Die Bewertung folgt den gemeinsamen Testbedingungen und Bewertungsverfahren von JVET für 360-Grad-Videos.Source: Kurzfassung · p. 9
• Die Hauptmethode projiziert JVET-Testsequenzen in ein Codierungsformat, komprimiert sie mit dem Netzwerk und konvertiert die Rekonstruktionen anschließend in das Quellprojektionsformat zurück.Die verschiedenen Projektionen werden anhand objektiver, von 360Lib berechneter Metriken verglichen.Source: Kurzfassung · p. 9

Symbols and Notations

• Optical-flow and frame notation includes v_t for optical flow, m_t for the quantized signal, reconstructed motion and frames, predicted frames, residual data, and displacement fields.The notation also defines pixel intensity, longitude, latitude, and 2D and 3D coordinate systems.Source: Symbols and Notations · p. 13
• Neural-compression notation includes convolution parameters and biases, downsampling and normalization factors, the loss L(.), l2 norm, posterior q, VAE parameters ϕ and θ, and KL divergence Dkl.The list also defines Charbonnier loss and latent-space variables.Source: Symbols and Notations · p. 13
• Geometric and coding notation includes frame dimensions, tangent functions, reference and coding faces, yaw, pitch, and roll rotations, distortion d(., .), Lagrange multiplier λ, frame size, spherical factor, and radius r.The rotation symbols are β for yaw, α for pitch, and γ for roll.Source: Symbols and Notations · p. 14

Abbreviations and Acronyms

• The abbreviations include projection formats such as ERP, PERP, CMP, EAC, HEC, ACP, and RSP, alongside codecs, models, and metrics used in the study.Key terms include PSNR, S-PSNR, WS-PSNR, BD Rate, DNN, CNN, VAE, DVC, HEVC, and JVET.Source: Abbreviations and Acronyms · p. 15

Introduction

• 360-degree video lets users look in all directions and is created by combining video from cameras arranged in a circle.It supports applications in entertainment, shopping, healthcare, and virtual reality.Source: Introduction · p. 17
• Unlike traditional video, 360-degree content surrounds users with 360-degree horizontal and 180-degree vertical perspectives.This expanded field of view enhances the viewing experience but increases processing difficulty.Source: Introduction · p. 17
• The growing demand for 360-degree content is accompanied by challenges in processing its high-resolution imagery.Source: Introduction · p. 17

Motivation

• The research is motivated by the large file sizes and transmission demands of immersive 360-degree video, building on prior projection-compression studies focused mainly on codecs such as HEVC.The introduction positions the work within broader efforts to improve omnidirectional video coding efficiency.Source: Chapter 1 Introduction · p. 18

Background

• Spherical 360-degree video must be projected onto a 2D plane because current video standards process flat rectangular frames.This projection enables existing video coding frameworks but introduces format-specific challenges.Source: Chapter 1 Introduction · p. 18
• Projection selection affects compression because formats such as ERP and CMP introduce different spatial distortions, including ERP oversampling near the poles.The introduction identifies projection choice as essential for effective compression of spherical content.Source: Chapter 1 Introduction · p. 18

Research objective and evaluation

• The study evaluates end-to-end optimized video compression networks with the scale-space flow network under JVET common test conditions.JVET testing sequences are converted to coding projections, compressed, reconstructed, returned to the source projection, and assessed with 360Lib metrics.Source: Chapter 1 Introduction · pp. 18–19

Proposed pipeline

• A three-layer trainable reprojection pipeline converts the reference projection, compresses the coding projection, and converts it back without learning parameters in the projection layers.The approach combines Python reprojection layers with the compression model to address projection-specific 360-degree video challenges.Source: Chapter 1 Introduction · p. 19

Learning-based Image and Video compression

• The chapter frames compression as reducing bit count while preserving quality by removing redundancy through transformation, prediction, quantization, and entropy coding.JPEG uses non-overlapping 8x8 block-DCT with DC prediction, while JPEG 2000 uses wavelets and EBCOT [20].Source: §2.1 · p. 21

2.1 Image compression

• Neural image compression replaces manually tuned components with end-to-end learned transforms, with Ballé et al.’s model outperforming JPEG2000 on PSNR and MS-SSIM.The framework uses analysis and synthesis transforms, GDN, uniform-noise quantization, and rate-distortion optimization.Source: §2.1 · pp. 22–23

2.2 Video compression

• Traditional video codecs use handcrafted components including block-based motion estimation and DCT, but lack end-to-end optimization.DVC uses learned models for motion estimation and compression, motion compensation, and residual transformation within an end-to-end framework.Source: §2.2 · pp. 24–27

2.3 Compression models

• Deep-learning video compression remains nascent, with no definitive optimal solution, and includes autoencoders, frame interpolation, and optical-flow motion compensation.Autoencoders learn lower-dimensional latent representations for reconstruction, while VAEs add probabilistic latent modeling and KL regularization [41] [43].Source: §2.3 · p. 27, §2.3.1 · pp. 27–29

2.3.1 Autoencoders

• Autoencoders compress by encoding inputs into constrained latent spaces and decoding them for reconstruction, preventing identity copying while potentially improving compression efficiency and image quality.VAEs further model posterior and likelihood distributions to generate similar images from compact representations [41] [43].Source: §2.3.1 · pp. 27–29

2.3.2 Frame interpolation

• Frame interpolation estimates optical flow and synthesizes missing frames by warping and blending temporal neighbors, but accuracy decreases as frame intervals increase.CNNs can predict flow and interpolate frames, while phase-based methods offer an alternative but may lose high-frequency details during substantial temporal changes.Source: §2.3.2 · pp. 29–31

2.3.3 Motion compensation via optical flow

• Optical-flow motion compensation reduces temporal redundancy by warping decoded frames, but occlusions, lighting, scene changes, and complex motion can cause inaccurate predictions.FlowNet2 improved optical-flow estimation over traditional approaches, whereas its memory demands motivated lightweight designs such as SpyNet.Source: §2.3.3 · pp. 31–32, §2.4 · p. 33

2.4 Scale-space flow network

• Scale-space flow adds an adaptive scale field to optical flow, reducing intermediate residual error and improving residual compression efficiency.The model uses scale-space warping with trilinear sampling and end-to-end latent reconstruction; evaluations reported superiority over H.264, HEVC, and leading learning-based methods.Source: §2.4 · pp. 33–34

360-degree Projection Formats

• 360-degree video requires 2D projection before conventional encoding, and projection choice affects redundancy, shape distortion, quality, and resource efficiency.The section introduces commonly used formats provided by the 360Lib app.Source: 360-degree Projection Formats · p. 35

3.1 Equi-rectangular projection format

• ERP uses a single face and maps 2D sampling coordinates to longitude, latitude, and spherical coordinates.Longitude and latitude are computed as ϕ = (u −0.5) · (2π) and θ = (0.5 −v) · π before obtaining (X, Y, Z).Source: §3.1 · p. 36

3.2 Padded Equi-rectangular projection format

• PERP duplicates samples across ERP’s discontinuous left and right boundaries to reduce seam artifacts in reconstructed viewports.Padding copies pixels from the left image edge to the right image edge.Source: §3.2 · p. 37

3.3 Cubemap projection format

• CMP projects the sphere onto six square faces, but its uneven sampling concentrates samples near edges and reduces coding efficiency.Several CMP variants were proposed to address this non-uniform distribution.Source: §3.3 · p. 38

3.4 Equi-angular cubemap projection format

• EAC modifies CMP coordinate mapping to produce a more uniform distribution and approximately equal angular spacing between samples on each face.The mapping functions are applied along both horizontal and vertical dimensions.Source: §3.4 · p. 39

3.5 Adjusted cubemap projection format

• ACP addresses CMP’s non-uniform sampling by using quadratic mapping functions, producing a distribution similar to EAC.The functions are applied in both horizontal and vertical dimensions.Source: §3.5 · pp. 39–40

3.6 Hybrid Equi-angular cubemap projection format

• HEC further modifies EAC with a new mapping function to address distortion of straight lines.The added function applies to the front, left, right, and back faces, while the u-axis remains unchanged.Source: §3.6 · pp. 40–41

3.7 Rotated sphere projection format

• RSP uses two face rows formed from an ERP image and a rotated ERP region, with approximately six percent redundant corner samples.360Lib implements RSP with six faces, deactivating overlapping samples to improve coding efficiency.Source: §3.7 · p. 42

4.1 360-degree content testing procedures

• The JVET procedure converts reference content into a coding projection, compresses it below the original quality, and converts the result back for fair comparison.This isolates compression performance from differences in source quality.Source: §4.1 · pp. 43–44, Figure 4.1 · p. 43

4.2 Implementation pipeline

• The implementation evaluates 8K YUV(4:2:0) JVET sequences in ERP across coding projections using eight scale-space-flow quality levels and BD-Rate.Performance is measured after inverse conversion using PSNR, WS-PSNR, S-PSNR-I, and Bjøntegaard Delta Bitrate.Source: §4.2 · p. 44, Table 4.1 · p. 45, Table 4.2 · p. 45

4.3 Projection Conversion using 360Lib

• 360Lib performs projection conversion face-by-face by mapping coding-plane samples through 3D XYZ coordinates to reference-plane samples with interpolation.The system also supports relative yaw, pitch, and roll rotations between reference and coding projections.Source: §4.3 · p. 46, §4.4 · p. 47

4.4 Compression model

• The compression model is the scale-space-flow model of E. Agustsson et al., using CompressAI pre-trained weights and RGB tensors shaped as (N, C, H, W).Inputs must lie in [0, 1], have dimensions of at least 64 pixels, and use power-of-two height and width.Source: §4.4 · p. 47
• Rate points are controlled by λ, with higher values reducing bitrate while increasing distortion and lower values producing the opposite trade-off.Separate models are optimized for MSE and MS-SSIM across nine levels spanning 0.025–0.8 bpp.Source: §4.4 · pp. 47–48

4.5 Evaluation metrics

• Because 360-degree content is spherical, evaluation uses conventional PSNR together with spherical metrics adapted to projection-dependent distortions.PSNR compares signal power with noise power and is commonly used for codec comparisons.Source: §4.5 · pp. 48–49, §4.5.1 · p. 48

4.5.1 Peak-Signal-to-Noise ratio

• PSNR measures reconstruction quality from pixelwise luminance error between compressed and reference frames using mean-squared error over the frame dimensions.The frame size is represented by (M, N).Source: §4.5.1 · p. 48, §4.5 · p. 49

4.5.2 Weighted-to-Spherically-Uniform PSNR

• WS-PSNR [79] weights projection samples using a spherical factor that varies by format, including ERP, CMP-based projections, ACP, and EAC.The factor accounts for spherical curvature and projection-specific geometric distortion.Source: §4.5.2 · pp. 49–50

4.5 Evaluation metrics

• The metric treatment accounts for projection-specific spherical factors, including EAC-equivalent factors on HEC top and bottom faces and active-sample-only factors for RSP.RSP excludes inactive sample positions created when its circular faces are collapsed.Source: §4.5 · p. 51

4.5.3 Spherical PSNR

• Spherical PSNR samples uniformly distributed sphere points, maps them to reference and compressed-frame positions, accumulates distortion, and derives PSNR.The mapping uses a 3D-to-2D coordinate transformation for each sampled sphere point.Source: §4.5.3 · pp. 51–52, Figure 4.3 · p. 52

4.6 Projection conversion using neural networks

• The neural projection model contains fixed preprocessing and postprocessing layers around a compression model, with no learnable parameters in the projection conversions.The first layer converts the reference projection to the coding projection, while the third reverses that conversion.Source: §4.6 · p. 52
• For ERP-to-CMP conversion, generated projection points are mapped between planes, rounded to pixel locations, and sampled using nearest-neighbor interpolation.The mapping is not one-to-one, so some ERP points may remain unmapped while multiple nearby points select the closest integer coordinate.Source: §4.6 · pp. 53–54
• The forward pipeline converts ERP frames to CMP, compresses CMP frames, converts them back to ERP, and computes evaluation metrics and reconstruction loss in ERP space.Preprocessing and postprocessing use analogous operations with reversed projection directions.Source: §4.6 · p. 53
• Backpropagation propagates ERP-frame loss gradients through postprocessing, compression, and preprocessing by the chain rule to update model weights.The backward pass treats each layer as a black box while transforming gradients between compressed ERP and CMP representations.Source: §4.6 · p. 54
• Because forward projection mapping is not one-to-one, the backward pass models CMP-to-ERP relationships using zero-or-one mapping gradients and associated weight vectors.The resulting input gradients are used to minimize loss with respect to the uncompressed ERP frames.Source: §4.6 · pp. 55–56

5.1.1 Projection conversion

• ERP and Padded ERP show visible blockiness at the lowest compression level, especially in the water and around the boat, while differences are less significant at the highest level.The observed artifacts result from compression-induced image-quality loss and distortion, and similar behavior holds across other projection formats.Source: §5.1.1 · p. 58, Figure 5.2 · p. 59

5.1.2 Projection compression

• ERP and Padded ERP outperform the other projections in end-to-end rate-distortion performance, while RSP and CMP have the lowest values.Cube map-based projections show similar performance, with EAC, HEC, and ACP between the ERP-based and lowest-performing formats.Source: Figure 5.5 · p. 61, §5.1.2 · p. 62, Figure 5.6 · p. 63

5.1.3 Projection comparison

• All bd-rate values for cube map-based and RSP-based projections are positive relative to ERP, indicating that ERP outperforms each of them.PERP performs closest to ERP and outperforms it in all sequences except Harbor and PoleVault; additional projection tests also preserve ERP and PERP’s advantage.Source: §5.1.3 · pp. 62–63, Table 5.1 · p. 63
• Hybrid HM-16.16 compression favors cubemap-based projections, whereas the scale-space flow model favors ERP-based projections because block coding accommodates faces while pixel-wise optical flow benefits from single-face continuity.CMP face discontinuities produce inconsistent motion-vector directions for pixel-wise optical flow, while block-based codecs can partition the multiple faces and use separate motion vectors.Source: §5.1.3 · p. 64, Table 5.2 · p. 65

5.2 Evaluation of the projection layer

• The fully end-to-end model successfully converts ERP to CMP, compresses the CMP frames, and converts them back to ERP, with gradients propagated through nearest-neighbor projection layers.Autograd validation confirms that the implemented backward computation works as mathematically specified.Source: §5.2 · pp. 65, 66, 68, Figure 5.7 · p. 66

5.3 Methods comparison

• For CMP, the 360Lib-based projection method achieves better rate-distortion performance than the model-based projection method for all sequences.The curves have similar shapes, but the model-based method yields lower end-to-end PSNR; differences are attributed to conversion-method discrepancies and ERP reference-frame handling.Source: Figure 5.8 · p. 68, §5.3 · p. 69, Table 5.3

Conclusion

• The study established a JVET-based evaluation pipeline using 360Lib objective metrics to assess projected 360-degree video compression.The pipeline evaluated different projections on JVET testing sequences according to common test conditions and evaluation procedures.Source: Conclusion · p. 71
• A three-layer model transformed the reference projection, compressed the coding projection, and converted the output back to the reference projection.The reprojection layers performed only input and output preprocessing and contained no learning parameters.Source: Conclusion · p. 71
• Learning-based compression favored ERP-based projections, while HM-16.16 favored CMP-based projections, revealing codec-dependent projection efficiency.The contrasting results reflect fundamental differences between learning-based and traditional video codecs.Source: Conclusion · p. 71
• Hybrid codecs suited CMP-based projections because their block-based architecture accommodates CMP’s multiple faces, improving compression performance and coding efficiency.This architectural explanation accounts for the traditional-codec advantage of CMP-based projections.Source: Conclusion · p. 71

• Single-face formats such as ERP achieved superior performance by producing smoother motion vectors and higher compression efficiency.Source: Chapter 6 Conclusion · p. 72
• Training learning-based models with data from multiple perspectives may improve handling of CMP by addressing multi-angle motion changes and enabling more effective compression strategies.The conclusion calls for further research into learning-based compression methods and their enhancements.Source: Chapter 6 Conclusion · p. 72
• 360lib-based projection outperformed model-based projection for CMP, with projection-conversion differences between ERP and CMP identified as the main cause of the end-to-end PSNR gap.The comparison covered different sequences and focused on CMP projection.Source: Chapter 6 Conclusion · p. 72
• The study provides insights into projection performance across traditional and learning-based codecs, supporting future research and development in 360-degree video compression.Source: Chapter 6 Conclusion · p. 72