PaperlayerView exact revision on arXiv

Source-linked AI summary

HEALPix-Based Implicit Neural Compression of Omnidirectional Images

Yulong Cheng, Youneng Bao, Junfeng Zhou, Mu Li, Jie Wen

arXiv:2610.32561v1cs.CV

TL;DR

Omnidirectional codecs need high compression efficiency and low decoding complexity, but ERP sampling oversamples polar regions and produces latitude-dependent receptive fields. This paper proposes an overfitted codec using hierarchical HEALPix latents, spherical context modeling, and multi-ring convolution. It reports viewport BD-rate reductions over VVC and Cool-Chic 5.0, alongside faster decoding than evaluated learned autoencoders.

  • Problem

    ERP-based omnidirectional coding has nonuniform spherical sampling and latitude-dependent receptive fields, while codecs must balance compression efficiency with low decoding complexity.

    Source: §I · pp. 1–2

  • Method

    The codec represents each image with hierarchical equal-area HEALPix latents, a spherical inter-scale entropy model, and multi-ring spherical convolution.

    Source: §I · p. 2

  • Results

    16.3%, 26.5%, and 19.5% BD-rate reductions over VVC under V-PSNR, V-MS-SSIM, and V-LPIPS, respectively; decoding is 2.6–3.6× faster than evaluated learned autoencoders.

    Source: §I · p. 2

  • Takeaways & Limitations

    The viewport-based evaluation reports a 13.1% BD-rate reduction over Cool-Chic 5.0 under V-PSNR, with faster decoding than the evaluated learned autoencoders.

    Source: §I · p. 2

    Entropy decoding within each level is serial and dominates decoding time; ERP-to-sphere resampling degrades the fitting target and limits achievable quality.

    Source: §V · p. 11

Abstract

from arXiv · show

Real-time decoding of omnidirectional images is essential for immersive virtual reality (VR) experiences. Compared with conventional transform-based learned image compression (LIC), overfitted codecs offer lower decoding complexity by representing each image with a lightweight implicit neural representation (INR). However, existing approaches typically operate on the equirectangular projection (ERP), which introduces nonuniform spherical sampling and spatial distortion, leading to redundant sampling near the poles and latitude-dependent receptive fields. To address these limitations, we propose a spherical overfitted codec based on a multi-scale Hierarchical Equal Area isoLatitude Pixelization (HEALPix) representation, which enables equal-area sampling and efficient hierarchical modeling of spherical signals. Building on this representation, we develop a Spherical Inter-Scale Conditional entropy model that exploits spatial dependencies and cross-scale correlations among HEALPix latents to improve coding efficiency. We further introduce a Multi-Ring Spherical Convolution that aggregates neighboring pixels over a nearly uniform spherical receptive field across latitudes. Experiments on a public omnidirectional image dataset demonstrate BD-rate reductions of 16.3%, 26.5%, and 19.5% over VVC in terms of V-PSNR, V-MS-SSIM, and V-LPIPS, respectively. Our method also achieves a 13.1% BD-rate reduction over ERP-based Cool-Chic 5.0 in terms of V-PSNR, while providing 2.6-3.6x faster decoding than the evaluated learned autoencoder codecs.

I. INTRODUCTION

Omnidirectional codecs must balance compression efficiency with low decoding complexity, but ERP-based overfitted codecs mismatch the sphere’s sampling and neighborhood geometry. The paper proposes a multiscale HEALPix overfitted codec with geometry-aware coding modules and reports viewport-quality and decoding gains.

I. INTRODUCTION

  • Overfitted codecs optimize a compact neural representation for each image, avoiding dependence on dataset generalization and enabling low-complexity decoding.The cited codecs require only a few thousand multiplications per pixel.Source: §I · p. 1
  • ERP-based overfitted codecs inherit planar assumptions that waste bits near the poles and make receptive fields shrink with latitude.At 88° latitude, a fixed planar kernel covers approximately 3.5% of its equatorial spherical footprint.Source: §I · p. 2
  • 16.3%, 26.5%, and 19.5% BD-rate reductions over VVC under V-PSNR, V-MS-SSIM, and V-LPIPS; 13.1% over Cool-Chic 5.0 in V-PSNR, with 2.6–3.6× faster decoding than learned autoencoders.The results use viewport-based evaluation.Source: §I · p. 2
  • The proposed multiscale HEALPix representation uses equal-area sampling and hierarchical latent grids to avoid polar oversampling and model content across scales.Its iso-latitude ring structure also supports efficient neighborhood access and structured decoding.Source: §I · p. 2
  • The Spherical Inter-Scale Conditional Entropy Model uses spherical-distance neighbors and coarser-scale ancestors, while Multi-Ring Spherical Convolution maintains nearly uniform receptive fields across latitudes.The entropy model exploits spatial and cross-scale dependencies among HEALPix latents.Source: §I · p. 2

A. Learned Image Compression

This related-work section reviews learned compression, INR-based overfitted codecs, and approaches to omnidirectional compression. It positions the paper’s HEALPix overfitted codec against dataset-trained codecs and prior spherical methods.

A. Learned Image Compression

  • Learned image codecs map images through analysis and synthesis transforms, quantize latents, and use entropy coding; context models improve compression but can require serial decoding.Other work changes transforms, entropy models, or training procedures to improve coding efficiency or reduce computational cost.Source: A. Learned Image Compression · p. 2
  • Dataset-trained autoencoders have computationally demanding decoders, while adapting an overfitted codec to each image can improve its rate–distortion performance.The passage contrasts modern autoencoders requiring about one million multiplications per pixel with lightweight overfitted codecs.Source: A. Learned Image Compression · pp. 2–3

B. INR-Based Image Compression

  • INR codecs represent images with neural functions or hierarchical latent grids; Cool-Chic extends the latter approach and this work extends its grids to HEALPix.Other INR work addresses fitting speed, weight compression, quantization, signal partitioning, or perceptual distortion.Source: B. INR-Based Image Compression · p. 3
  • Prior omnidirectional INR applications include HEALPix signal fitting and ERP-frame video, but the reviewed methods do not combine spherical entropy coding with rate–distortion optimization.The cited examples include weather and cosmic microwave background maps, as well as 360° and multi-view video.Source: B. INR-Based Image Compression · p. 3

C. Omnidirectional Image Compression

  • Omnidirectional compression adapts projections or networks to spherical geometry, or avoids projection with spherical grids such as HEALPix; reviewed HEALPix codecs remain large, dataset-trained autoencoders.The paper’s codec places its latent grids, entropy model, and synthesis network on equal-area HEALPix pixels, with a decoder of only a few thousand weights.Source: C. Omnidirectional Image Compression · p. 3

III. PROPOSED METHOD

The method represents each panorama with hierarchical latent grids on HEALPix and uses the grid’s geometry to structure sampling and cross-scale relationships. The section introduces the representation and outlines how the codec fits and decodes it.

III. PROPOSED METHOD

  • The codec maps an ERP-stored omnidirectional image onto the unit sphere and represents it with hierarchical HEALPix latent grids.The overview describes entropy-model rate estimation and coarse-to-fine reconstruction from quantized latents.Source: A. Spherical Latent Representation · pp. 3–4, §III · p. 3

A. Spherical Latent Representation

  • Each latent grid level refines the previous resolution, with coarse levels capturing global structure and finer levels adding local texture.The main experiments use eight levels, starting at Nside = 2; the high-resolution experiment starts at Nside = 4.Source: A. Spherical Latent Representation · p. 4
  • HEALPix sampling regions have equal spherical area, avoiding grid-induced redundant polar rate; neighbor-distance variability is reported as 11.6% for HEALPix versus 18.1% for ERP.The table caption identifies the comparison as relative standard deviation of neighbor distances across the sphere.Source: A. Spherical Latent Representation · p. 4, Table I · p. 4
  • Each refinement splits a sampling region into four equal-area regions, forming a quad-tree with a unique parent for every sampling point.The parent–child relation provides an exact correspondence between levels.Source: A. Spherical Latent Representation · p. 4

B. Quantization

The codec quantizes latents and decoder weights differently, then uses the SISC entropy model to predict each latent’s probability from spherical same-scale and cross-scale contexts.

B. Quantization

  • Latents are rounded to the nearest integer after fitting, with soft rounding and additive noise early in fitting, then straight-through gradients later.The differentiable approximation is gradually reduced so latents approach integers.Source: B. Quantization · p. 4
  • Decoder weights are quantized using module-wise steps and encoded with Exp-Golomb codes, with each module’s step and code order selected by rate–distortion search after training.Weights remain floating point during training.Source: B. Quantization · p. 5

C. Spherical Inter-Scale Conditional Entropy Model

  • SISC combines decoded same-grid neighbors and coarser-grid ancestors, selected using spherical distance and the HEALPix quad-tree, to predict one Laplace distribution per quantized latent.Fig. 3 depicts the two contexts and their shared predictor.Source: C. Spherical Inter-Scale Conditional… · p. 5, Figure 3 · p. 5

1) Intra-Scale Context:

The SISC model forms causal contexts from nearby decoded latents and their decoded ancestors, then predicts distribution parameters from both contexts.

1) Intra-Scale Context:

  • For each target latent, the intra-scale context selects its K already decoded same-grid neighbors by spherical distance, giving a similarly sized context region across latitudes.Early positions with fewer than K decoded neighbors use zeros for missing entries.Source: 1) Intra-Scale Context · p. 5

2) Inter-Scale Context:

  • Because decoding proceeds coarse to fine, the inter-scale context uses decoded ancestors that cover the target’s region at each coarser level.HEALPix’s quad-tree provides a unique ancestor chain; a level-specific linear layer maps those ancestors to an F-dimensional context, while the coarsest level uses zero.Source: 2) Inter-Scale Context · p. 5

3) Shared Predictor:

  • A shared predictor combines the intra- and inter-scale contexts to predict the location and raw log-scale parameters of a Laplace distribution.The predictor uses a linear layer and an MLP with ReLU activations and residual connections.Source: 3) Shared Predictor · pp. 5–6

4) Single-Laplace Model:

The entropy model assigns probabilities to integer-valued quantized latents using a Laplace distribution, and the decoder reconstructs the image from decoded latents.

4) Single-Laplace Model:

  • Each quantized latent is modeled by one Laplace distribution, and its probability is the distribution’s mass over that integer’s quantization bin.The latent rate is the total code length of the quantized latents.Source: 4) Single-Laplace Model · p. 6

D. Decoder: From Latents to the Image

  • The decoder entropy-decodes the latent grids with SISC, then reconstructs RGB values on the finest HEALPix grid.Its pipeline includes spherical upsampling to merge levels into a dense finest-grid representation, followed by a synthesis network that maps the representation to RGB.Source: D. Decoder: From Latents to the Image · p. 6

1) Multi-Ring Spherical Convolution:

Multi-Ring Spherical Convolution expands spatial context over HEALPix neighbors, while spherical upsampling merges hierarchical features onto the finest grid.

1) Multi-Ring Spherical Convolution:

  • Multi-Ring Spherical Convolution covers every sampling point within R hops on the HEALPix 8-neighbor graph, with kernel weights growing with covered area.For R = 3, the neighborhood contains 49 sampling points, whereas a dilated kernel always samples 9.Source: 1) Multi-Ring Spherical Convolution · p. 6

2) Spherical Upsampling:

  • The spherical upsampler processes levels coarse to fine, refining each quantized grid with a residual block before merging it into the finer feature stream.The refinement uses a radius-one depthwise multi-ring convolution, GELU, and pointwise convolution.Source: 2) Spherical Upsampling · p. 6
  • At each upsampling step, each fine point and its eight neighbors map to their parents, whose features are depthwise-weighted and concatenated with that level’s mixed features.A missing neighbor at an adjacency defect is replaced by the center point.Source: 2) Spherical Upsampling · p. 6
  • After the final step, the dense representation has N+1 channels, each carrying one level’s mixed features upsampled to the finest grid.The per-level mixers and upsampling filters form the spherical upsampler.Source: 2) Spherical Upsampling · p. 7

3) Synthesis Network:

The synthesis network builds spherical context through multi-ring convolutions before mapping features to RGB. Per-image fitting jointly optimizes codec parameters for spherical distortion and bitstream size, then quantizes decoder weights and fine-tunes latents.

3) Synthesis Network:

  • Multi-ring residual blocks widen the spherical receptive field from nearest neighbors to larger regions, while residual connections preserve local details.Pointwise convolutions project features, mix channels, and map the final features to RGB.Source: 3) Synthesis Network · p. 7

E. Rate–Distortion Optimization

  • All latents and decoder modules are fitted jointly per image to trade off decoded-sphere quality against bitstream size.The objective combines spherical distortion with latent and quantized decoder-weight rates; dividing by HW expresses the rates in bpp.Source: E. Rate–Distortion Optimization · p. 7
  • Distortion is evaluated over a nearly uniform Fibonacci lattice, whose unweighted average approximates squared error integrated over the sphere without latitude weighting.The lattice samples differ from the viewport rays used for evaluation.Source: E. Rate–Distortion Optimization · p. 7
  • SOAP optimizes the latents and decoder weights before weight quantization; the latents are then fine-tuned with quantized weights fixed.Decoder quantization step sizes and Exp-Golomb orders are selected by minimizing the rate–distortion objective.Source: E. Rate–Distortion Optimization · p. 7

IV. EXPERIMENTS

The experiments compare the codec quantitatively and visually with traditional, learned, and INR codecs, then report ablations and complexity analysis. The supplied figures show rate–distortion curves for a 100-image benchmark and a higher-resolution evaluation.

IV. EXPERIMENTS

  • The experimental section compares traditional, learned, and INR codecs quantitatively and visually, then reports ablations and complexity analysis.The section begins with the experimental setup.Source: §IV · p. 7
  • Fig. 6 presents rate–distortion curves on the 100-image benchmark under V-PSNR, V-MS-SSIM, and V-LPIPS across six viewports.C5 denotes Cool-Chic 5.0.Source: Figure 6 · p. 8
  • Fig. 7 presents rate–distortion curves at 1536×3072 on 99 images, computed on six 1024×512 viewports.LPIPS uses AlexNet; C5 denotes Cool-Chic 5.0.Source: Figure 7 · p. 8

A. Experimental Setup

Experiments use two omnidirectional-image benchmarks and evaluate all codecs on six rendered viewports. Comparisons use viewport quality metrics and each codec’s actual bitstream rate across traditional, learned, and INR baselines.

A. Experimental Setup

  • The main benchmark contains 100 images at 768×1536, with a second set of 99 panoramas at 1536×3072 for higher-resolution testing.Both datasets are evaluated using the viewport metrics described in the paper.Source: A. Experimental Setup · p. 8
  • Six rectilinear viewports—four equatorial and two polar—are rendered along identical viewing rays, with each reconstruction read directly from its decoded format.Each codec undergoes one mapping from its decoded format to the viewports.Source: A. Experimental Setup · p. 8
  • Table II reports BD-rate against each baseline under three viewport metrics, while Table III reports equal-quality rate changes and decoder multiplications per pixel for ablations.Negative BD-rate values in Table II mean fewer bits at the same quality; positive Table III values mean more bits.Source: Table II · p. 8, Table III · p. 9
  • Quality is compared using V-PSNR, V-MS-SSIM, and V-LPIPS, while rate is the size of each codec’s actually produced bitstream.Overfitted INR bitstreams include coded latents, quantized decoder weights, and required header data; learned-codec bitstreams contain coded latents.Source: A. Experimental Setup · pp. 8–9
  • The comparison includes VVC, three learned planar codecs, two planar INR codecs, and two learned omnidirectional codecs.OSLO-IC was trained from released code because no trained model was available; all codecs are compared in the viewport domain.Source: A. Experimental Setup · p. 9

B. Rate–Distortion Performance

The codec reduces rate against VVC across three viewport metrics and retains an advantage at higher resolution. Ablations show contributions from hierarchical HEALPix representation, inter-scale entropy context, and multi-ring spatial convolution.

B. Rate–Distortion Performance

  • 16.3%, 26.5%, and 19.5% lower BD-rate than VVC under V-PSNR, V-MS-SSIM, and V-LPIPS, respectively.Against MLIC++, savings are 5.5%, 12.5%, and 12.8%; against OSLO-IC, they are 27.8%, 12.2%, and 42.0%, respectively.Source: B. Rate–Distortion Performance · p. 9
  • At higher resolution, BD-rate remains negative against every baseline under all three metrics, so the advantage holds at that resolution.The comparison is repeated at 1536×3072.Source: B. Rate–Distortion Performance · p. 9
  • At similar rates on one panorama, five of six baselines use 3–55% more bits than our codec, and all six have 0.45–1.09 dB lower V-PSNR.OSLO-IC uses 1.7% fewer bits; the figure shows six viewports and six crops.Source: B. Rate–Distortion Performance · p. 9, Figure 8 · p. 10

C. Ablation Studies

  • 44.5% more bits on HEALPix and 57.6% more on ERP are required when using a single level; with eight levels, ERP costs 4.2% more bits and 6928 rather than 4619 MAC/px.Coarse levels encode low-frequency structure at a much lower rate; the HEALPix grid also spends 14.7% of bits on the 13.4% of sphere above |δ|=60°, versus 27.9% for ERP.Source: C. Ablation Studies · pp. 9, 11, Figure 10 · p. 11
  • Without entropy context, rate increases by 48.4%; inter-scale context reduces rate by 2.3% at a cost of 1100 MAC/px.Using RING-order rather than nearest spherical neighbors costs 7.1% more bits, and retaining only the parent costs 5.0% more; Fig. 9 shows a 29% total-rate rise without inter-scale context at similar V-PSNR.Source: Figure 9 · p. 10, C. Ablation Studies · p. 11
  • Removing spatial convolution increases rate by 7.2%; at equal complexity, dilated and single-ring alternatives cost 2.1% and 1.6% more bits, respectively.Sharing one weight per ring costs 1.5% more bits under V-PSNR and 5.3% more under V-LPIPS; the authors attribute this to the importance of neighbor direction for oriented structures.Source: C. Ablation Studies · p. 11
  • Replacing SOAP with Adam increases rate by 0.6% under the same iteration count, while decoding complexity remains unchanged.SOAP mainly accelerates convergence and shortens fitting time.Source: C. Ablation Studies · p. 11

D. Complexity Analysis

The codec reports bitrate savings over VVC and faster decoding than compared learned autoencoders. Its timing setup and remaining decoding and resampling constraints qualify those results.

D. Complexity Analysis

  • Decoding times use one CPU core on an Intel Xeon Silver 4310 at batch size 1, with model building and checkpoint loading excluded.LIC360 is timed on one NVIDIA RTX 4090 because its released code runs only on a GPU, so those times are not directly comparable.Source: D. Complexity Analysis · p. 11

V. CONCLUSION AND DISCUSSION

  • 16.3%, 26.5%, and 19.5% bit-rate savings over VVC under V-PSNR, V-MS-SSIM, and V-LPIPS, respectively; decoding is 2.6–3.6× faster than compared learned autoencoders.The codec also outperforms all compared codecs.Source: §V · p. 11
  • Entropy decoding is serial within each level and dominates decoding time; ERP inputs must also be resampled onto the sphere, degrading the target, with that loss included in reported results.Parallel decoding of sampling-point groups and datasets stored natively in HEALPix are identified as future work.Source: §V · p. 11

Related papers

Loading 2610.32561v1…