PaperlayerView exact revision on arXiv

Source-linked AI summary

HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation

Ho Man Kwan, Ge Gao, Fan Zhang, Andrew Gower, David Bull

arXiv:2306.09818v3eess.IVcs.CV

TL;DR

Existing INR-based video codecs have not achieved rate-distortion performance comparable to state-of-the-art codecs. HiNeRV combines hierarchical encoding with a unified frame-and-patch representation and a refined compression pipeline. It improves on INR baselines and is competitive with other codecs, including reported gains over HNeRV and DCVC on UVG.

  • Problem

    Existing INR-based video codecs have not matched state-of-the-art conventional and learning-based codecs in rate-distortion performance, while common architectures constrain representation capacity.

    Source: §1 · pp. 1–2

  • Method

    HiNeRV uses hierarchical encoding with bilinear interpolation and local feature grids, supports both frame-wise and patch-wise representation, and refines compression with adaptive pruning and quantization-aware training.

    Source: §1 · p. 2

  • Results

    HiNeRV reports a 72.3% overall BD-rate gain over HNeRV on UVG, measured in PSNR, and is competitive with learning-based codecs, including 43.4% over DCVC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate the potential of INRs for video compression, despite entropy coding and quantization not being optimized in the loop.

    Source: §5 · p. 10

    Full end-to-end optimization remains future work, particularly incorporating entropy coding and quantization.

    Source: §5 · p. 10

Abstract

from arXiv · show

Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. This is mainly due to the simplicity of the employed network architectures, which limit their representation capability. In this paper, we propose HiNeRV, an INR that combines light weight layers with novel hierarchical positional encodings. We employs depth-wise convolutional, MLP and interpolation layers to build the deep and wide network architecture with high capacity. HiNeRV is also a unified representation encoding videos in both frames and patches at the same time, which offers higher performance and flexibility than existing methods. We further build a video codec based on HiNeRV and a refined pipeline for training, pruning and quantization that can better preserve HiNeRV's performance during lossy model compression. The proposed method has been evaluated on both UVG and MCL-JCV datasets for video compression, demonstrating significant improvement over all existing INRs baselines and competitive performance when compared to learning-based codecs (72.3% overall bit rate saving over HNeRV and 43.4% over DCVC on the UVG dataset, measured in PSNR).

1 Introduction

Existing INR-based video codecs have not matched conventional or learning-based codecs, in part because their architectures limit representation capacity. HiNeRV combines hierarchical encoding, a unified frame-and-patch representation, and a refined compression pipeline, achieving strong results against INR baselines and competitive results against other codecs.

1 Introduction

  • Existing INR codecs remain inferior to state-of-the-art standard and learning-based codecs, while conventional convolutional layers and Fourier-based positional encoding constrain capacity and encoding speed.The paper identifies parameter inefficiency and long training time as limitations of common INR designs.Source: §1 · pp. 1–2
  • Hierarchical encoding with bilinear interpolation and local feature grids replaces less parameter-efficient upsampling, enabling deeper and wider networks with greater representation capacity.The architecture primarily uses MLPs and depth-wise convolutions rather than dense convolutions.Source: §1 · p. 2
  • Training with overlapped patches lets HiNeRV switch between frame-wise and patch-wise representations, improving performance over either setting and offering compute-versus-memory flexibility.Frames can optimize computational complexity, while patches can minimize memory footprint.Source: §1 · p. 2
  • Adaptive pruning and quantization-aware training refine model compression to better preserve reconstruction quality and support lower-bit-depth quantization with an improved rate-distortion trade-off.The paper applies quantization-aware training before quantization.Source: §1 · pp. 2–3
  • 72.3% overall BD-rate gain over HNeRV on UVG, measured in PSNR; HiNeRV is also competitive with codecs, including 43.4% over DCVC.The paper reports that HiNeRV was not fully end-to-end optimized for video compression.Source: §1 · p. 2

2 Related work

Related work covers learning-based video codecs and implicit neural representations, including NeRV-style approaches. It describes both their progress and remaining limits in complexity and rate-distortion performance.

2.1 Video compression

  • Learning-based codecs replace modules in conventional pipelines or use end-to-end coding architectures, but can be computationally complex and decode more slowly than conventional codecs.The paper notes that this can render them impractical for real-time applications, especially for high-quality, high-resolution video.Source: §2.1 · p. 3

2.2 Implicit neural representation

  • INRs use positional encoding to map coordinates to signal values; periodic functions support high-frequency learning, while grid features address slow convergence and improve reconstruction quality.The cited work applies INRs to images, videos, and volumetric content.Source: §2.2 · p. 3
  • NeRV recasts video INRs as frame-wise representations with fast decoding, inspiring CNN-based frame- and patch-coordinate methods used across several video tasks.Applications listed include denoising, interpolation, inpainting, super-resolution, and compression.Source: §2.2 · p. 3
  • NeRV and related INR codecs compress models through pruning, quantization, and entropy coding, but have not matched conventional or learning-based codecs in rate-distortion performance.The paper characterizes their decoding capability as fast despite this rate-distortion gap.Source: §2.2 · p. 3

3 Method

HiNeRV represents video through coordinate-conditioned feature maps that are progressively upsampled and processed into reconstructed patches. Its method combines grid-based encodings, hierarchical local encodings, flexible frame/patch computation, and a refined compression pipeline.

3 Method

  • HiNeRV treats video encoding as coordinate-to-video regression and reconstructs each M × M patch with a base layer, stem, progressive blocks, and a head.The coordinate input is (i, j, t), and the final block's feature maps are projected to the output patch.Source: §3 · pp. 3–4

3.1 Base encoding and stem

  • Grid-based positional features are interpolated using frame coordinates and time, then projected by a stem convolution into initial feature maps.The multi-resolution temporal grids increase channel count as temporal resolution decreases to maintain a compact grid representation.Source: §3.1 · p. 4

3.2 HiNeRV block

  • Each HiNeRV block bilinearly upsamples its input maps, adds projected hierarchical encoding, and processes the result into transformed feature maps.After the first block, the first layer reduces channels to save computation on high-resolution maps.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 Head layer

  • A linear layer with sigmoid activation maps the final block features to the reconstructed output patch.The head applies Fhead to XN to produce Y.Source: §3.3 · p. 5

3.4 Upsampling with the hierarchical encoding

  • Hierarchical encoding adds local positional information to bilinearly upsampled maps while using smaller grids than global-coordinate encoding.The local coordinates are relative to the nearest original-map pixel; temporal local grids interpolate using those coordinates and frame index.Source: §3.4 · pp. 5–6

3.5 Unifying frame-wise and patch-wise representations

  • Overlapped, padded patches let HiNeRV switch between frame-wise and patch-wise representation while producing the same output in either configuration.Padding also enables parallel patch computation and random patch sampling during training, though it adds computational overhead.Source: §3.5 · p. 6

3.6 The model compression pipeline

  • The refined compression pipeline combines layer-adaptive magnitude pruning with Quant-Noise quantization-aware fine-tuning to improve the rate-distortion trade-off at 6-bit quantization.Pruning weights neuron importance using both its L1 norm and layer size makes narrow layers less likely to be pruned.Source: §3.6 · p. 7

4 Experiments

Across video representation, compression, and ablation tests, HiNeRV improves reconstruction quality, achieves competitive codec performance, and outperforms the tested design variants.

4.1 Video representation

  • The video-representation benchmarks matched model parameter counts at each scale and used 300 training epochs with batch size one.They reported encoding and decoding speed in frames per second on an A100 GPU; HiNeRV and PS-NeRV used sampled patches during training.Source: §4.1 · p. 7
  • HiNeRV outperforms all benchmarked models in reconstruction quality at each scale on Bunny and UVG, including every UVG test sequence in PSNR.On ReadySetGo, it improves by more than 7.9 dB over NeRV.Source: §4.1 · p. 7, Table 1 · p. 8, Table 2 · p. 8
  • HiNeRV's encoding and decoding are slower than those of other methods, despite higher overall PSNR with fewer MACs.At 37 epochs, HiNeRV achieves reconstruction quality similar to HNeRV at 300 epochs on both datasets.Source: §4.1 · p. 8

4.2 Video compression

  • Compression benchmarks trained each model separately per video and applied 15% pruning, 60 fine-tuning epochs, then 30 epochs of Quant-Noise at a 90% noise ratio.Arithmetic entropy coding produced the actual rate, with pruning masks and quantization parameters included in bitstreams.Source: §4.2 · p. 8
  • HiNeRV is competitive on UVG and MCL-JCV, outperforming x265, DCVC, and VCT on PSNR and H.265 HM on MS-SSIM.The codec is not fully end-to-end optimized: entropy coding and quantization are not optimized in the training loop.Source: §4.2 · pp. 8–9, Table 3 · p. 9

4.3 Ablation study

  • The full HiNeRV model outperforms all alternatives V1–V11 in overall reconstruction quality on UVG.The ablations cover alternative upsampling layers and encodings, blocks, frame- and patch-wise configurations, and nearest-neighbor interpolation.Source: §4.3 · pp. 9–10, Table 4 · p. 10

5 Conclusion

HiNeRV combines hierarchical encoding, unified frame-and-patch representation, and a refined compression pipeline, achieving strong codec performance while leaving end-to-end optimization for future work.

5 Conclusion

  • The proposed codec's improvements are associated with bilinear-interpolation-based hierarchical encoding, unified representation, and a refined model-compression pipeline.Source: §5 · p. 10
  • HiNeRV achieves performance comparable to state-of-the-art end-to-end optimized learning codecs and significantly improves over NeRV-based algorithms, despite incomplete end-to-end optimization.The paper also reports that it is the first INR-based video codec to outperform HEVC HM Random Access mode on MS-SSIM.Source: §5 · p. 10
  • Future work should incorporate entropy coding and quantization to achieve full end-to-end optimization.Source: §5 · p. 10

A Implementation details

The implementation uses PyTorch-based tooling, original or reimplemented baselines, half-precision training with a stated exception, and specified complexity and codec-rate procedures.

A Implementation details

  • HiNeRV and other NeRV-based models were implemented in PyTorch with PyTorch Image Models, and torchac performed arithmetic coding.Source: A Implementation details · p. 14
  • The authors used original implementations for most NeRV baselines, reimplemented PS-NeRV, and reported HNeRV in full precision after half-precision hurt some MCL-JCV sequences.Source: A Implementation details · p. 14
  • Model complexity was estimated with DeepSpeed, while conventional codecs were tested at multiple QP values to obtain different rates.The authors note that some benchmark configurations' minimum network widths increased MACs despite relatively small model sizes.Source: A Implementation details · p. 14

B Comparison to other learning-based codecs

HiNeRV is slower to encode and decode than other INR-based methods, but decodes faster than the cited learning-based codecs. The supplementary material describes frame- and patch-wise configurations, model settings, and additional dataset results.

B Comparison to other learning-based codecs

  • HiNeRV (scale L) decodes at 10.9 FPS, versus 0.03/1.90 FPS for DCVC/DCVC-HEM on the same GPU.The DCVC/DCVC-HEM rates are reported for 1080p videos.Source: B Comparison to other learning-based… · p. 14
  • Encoding a 1080p video with 600 frames takes around 6.5/7.7/11.9 hours for HiNeRV scales S/M/L, respectively.The authors note encoding time can be reduced while retaining superior reconstruction quality.Source: B Comparison to other learning-based… · p. 14
  • HiNeRV performs better against learning-based codecs on UVG than on MCL-JCV, possibly because UVG sequences contain more frames.The authors identify this dataset difference as a limitation and suggest investigating it in future work.Source: B Comparison to other learning-based… · p. 14

C Details for unifying frame-wise and patch-wise representations (Sec. 3.5)

  • For patch-wise inference, padding and overlapped feature-map patches address boundary effects; padding adds computation but can improve performance.The described scale L example uses 2 × 2 base encodings without padding and 8 × 8 with padding, while frame-wise mode uses 18 × 32.Source: C Details for unifying frame-wise and… · pp. 14–15

D Configurations for HiNeRV; E.1 Video representation; E.2 Video compression

  • HiNeRV uses four blocks with reduction factor R = 2.0; configurations are specified for UVG and MCL-JCV, with additional Bunny and UVG representation results and UVG/MCL-JCV compression figures.The supplementary results include MS-SSIM tables for Bunny and UVG and compression figures that add FFNeRV and B-EPIC comparisons.Source: D Configurations for HiNeRV · pp. 15–16, Table 5 · p. 16, Table 6 · p. 16

E.3 Ablation study on the refined training pipeline

The supplementary ablations compare pruning and quantization choices in the refined training pipeline. Adaptive pruning performs best among the tested pruning approaches, and Quant-Noise without STE supports a 6-bit setting with bitrate savings over 8-bit quantization.

E.3 Ablation study on the refined training pipeline

  • The additional experiments use the same settings as the main paper's Section 4.3.Source: §E.3 · p. 17
  • The proposed adaptive pruning method outperforms standard pruning with or without head-weight exclusion, especially at larger pruning ratios.Standard pruning can remove a larger portion of head weights, from which performance may not recover even after fine-tuning.Source: §E.3 · pp. 17–18, Table 8 · p. 18
  • 6-bit Quant-Noise without STE provides relatively good performance and up to 25% bitrate savings compared with the commonly used 8-bit alternative.The comparison covers quantization with and without QAT, STE, and Quant-Noise.Source: §E.3 · p. 18, Table 9 · p. 18

E.4 Faster variants of HiNeRV

The faster HiNeRV-A and HiNeRV-B variants reduce computational cost while retaining competitive reconstruction quality. HiNeRV-B uses fewer operations than a twice-larger HNeRV and achieves around 80% of its decoding speed.

E.4 Faster variants of HiNeRV

  • The variants target lower MACs and faster encoding and decoding, with only a small drop in reconstruction quality.Source: §E.4 · p. 18
  • HiNeRV-A reduces lower-level feature-map size, while HiNeRV-B additionally reduces high-level blocks and removes normalization after upsampling.The variants adjust width to maintain network size at each scale.Source: §E.4 · p. 18
  • Compared with HNeRV at twice the size, HiNeRV-B achieves better average reconstruction quality with half the size, nearly one quarter the operations, faster encoding, and around 80% of HNeRV decoding speed.The authors note that further optimization may improve HiNeRV-B decoding speed.Source: §E.4 · p. 18, Table 11 · p. 19
  • HiNeRV-A reduces operations by up to 1/3 while retaining performance comparable to original HiNeRV, with fewer MACs than same-scale HNeRV.The comparison covers the faster variants against NeRV, HNeRV, FFNeRV, and original HiNeRV.Source: §E.4 · p. 18, Table 10 · p. 19, Table 11 · p. 19

E.5 Discussion regarding the ablation studies

The ablations favor hierarchical encoding and the unified representation, while also identifying sequence-specific exceptions and qualitative differences. Supplementary comparisons describe patch-boundary artifacts and visual detail at reduced bitrate.

E.5 Discussion regarding the ablation studies

  • Hierarchical encoding gives the largest boost on Jockey and ReadySetGo, with 0.78/1.06 dB PSNR improvements, while Fourier encoding wins on ShakeNDry.The authors suggest ShakeNDry's small-particle motion may suit Fourier encoding's absolute positional information.Source: §E.5 · p. 19
  • MLP-based HiNeRV drops from 34.69 dB to 32.16 dB average PSNR without hierarchical encoding, and no longer outperforms NeRV and HNeRV.HiNeRV with a ConvNeXt block also outperforms the MLP-block and normal-convolution variants.Source: §E.5 · p. 19
  • Frame-wise and patch-wise settings reduce coding performance versus the unified representation, especially on Jockey and ReadySetGo.The respective decreases are 0.25/0.49 dB for frame-wise and 1.90/1.92 dB for patch-wise settings.Source: §E.5 · p. 20
  • Replacing bilinear with nearest-neighbor interpolation causes only a small performance reduction; sub-pixel convolution alternatives do not match the proposed upsampling layer.The 3 × 3 sub-pixel variant only matches HNeRV on average, while the 1 × 1 variant improves but remains behind HiNeRV.Source: §E.5 · pp. 19–20

E.6 Qualitative comparison between different types of representation

  • Patch-wise outputs show boundary artifacts absent from the unified representation's outputs, even when both use patch-wise computation.The unified representation can also be applied to overlapped patches or whole video frames.Source: §E.6 · p. 20, Figure 7 · p. 20

E.7 Qualitative comparison between NeRV, HNeRV and HiNeRV

  • At approximately half the bitrate of NeRV and HNeRV, HiNeRV preserves more detail from the original video frames.The comparison uses the models presented for the main paper's video-compression section.Source: §E.7 · p. 20

F Limitations

HiNeRV has slow high-quality encoding and a larger memory footprint than other INR methods, though patch operations can reduce memory demand and improve parallelism.

F Limitations

  • Compressing a short video sequence at high quality takes multiple hours, and faster training is identified as future work.The paper notes this slow encoding speed is also a problem for existing INR-based methods.Source: F Limitations · p. 20
  • HiNeRV's more complex network has a larger memory footprint than other INR methods, but patch operations can reduce memory and improve parallelism.Source: F Limitations · p. 20
Loading 2306.09818v3…