PaperlayerView exact revision on arXiv

Source-linked AI summary

HNeRV: A Hybrid Neural Representation for Videos

Hao Chen, Matt Gwilliam, Ser-Nam Lim, Abhinav Shrivastava

arXiv:2304.02633v1cs.CV

TL;DR

Video storage and transfer remain costly, while implicit video representations are limited by fixed, content-agnostic embeddings. HNeRV combines content-adaptive embeddings with a redesigned decoder, improving regression performance and supporting efficient decoding, compression, and inpainting.

  • Problem

    Video storage, transfer, and processing are costly, while prior implicit representations rely on content-agnostic embeddings that limit regression capacity and internal generalization.

  • Method

    HNeRV stores videos using a learnable encoder for content-adaptive frame embeddings and a decoder with evenly distributed parameters.

  • Results

    +4.7 PSNR and 16× faster convergence than implicit methods, with improvements in video regression and internal generalization.

  • Takeaways & Limitations

    HNeRV provides simple, fast, flexible decoding and shows good performance for video compression and video inpainting.

  • Takeaways & Limitations

    HNeRV requires training for each new video, and selecting embedding size, model size, and architecture remains an open problem.

Abstract

from arXiv · show

Implicit neural representations store videos as neural networks and have performed well for various vision tasks such as video compression and denoising. With frame index or positional index as input, implicit representations (NeRV, E-NeRV, \etc) reconstruct video from fixed and content-agnostic embeddings. Such embedding largely limits the regression capacity and internal generalization for video interpolation. In this paper, we propose a Hybrid Neural Representation for Videos (HNeRV), where a learnable encoder generates content-adaptive embeddings, which act as the decoder input. Besides the input embedding, we introduce HNeRV blocks, which ensure model parameters are evenly distributed across the entire network, such that higher layers (layers near the output) can have more capacity to store high-resolution content and video details. With content-adaptive embeddings and re-designed architecture, HNeRV outperforms implicit methods in video regression tasks for both reconstruction quality ($+4.7$ PSNR) and convergence speed ($16\times$ faster), and shows better internal generalization. As a simple and efficient video representation, HNeRV also shows decoding advantages for speed, flexibility, and deployment, compared to traditional codecs~(H.264, H.265) and learning-based compression methods. Finally, we explore the effectiveness of HNeRV on downstream tasks such as video compression and video inpainting. We provide project page at https://haochen-rye.github.io/HNeRV, and Code at https://github.com/haochen-rye/HNeRV

1. Introduction

HNeRV addresses the limited regression capacity and interpolation generalization of content-agnostic implicit video representations with content-adaptive embeddings and a rebalanced decoder. It improves video regression and offers a simple representation for decoding, compression, and inpainting.

  • 1. Introduction: Implicit video representations use fixed spatial or temporal embeddings, which limit meaningful frame interpolation and force the decoder to store all visual information.These methods are simple and compact, but their positional embeddings provide no visual prior.
  • 1. Introduction: Content-adaptive embeddings provide visual information to the decoder, while HNeRV blocks distribute parameters more evenly toward later, high-resolution stages.The encoder makes HNeRV hybrid by storing tiny frame embeddings alongside a learned decoder.
  • 1. Introduction: HNeRV provides decoding advantages in speed, flexibility, and deployment compared with traditional codecs and prior learning-based approaches.Its non-autoregressive frame access supports efficient random access and parallel decoding.
  • 1. Introduction: HNeRV remains viable for video compression and shows promising video restoration performance, including verification on video inpainting.The paper also describes compression of the encoder and decoder components.
  • 1. Introduction: +4.7 PSNR and 16× faster convergence summarize HNeRV’s reported regression gains over implicit methods.The method also improves internal generalization.

2. Related Work

Related work spans implicit neural representations, neural video compression, model compression, and internal-learning approaches to video inpainting. HNeRV builds on these directions while combining video-specific representation with compression and restoration uses.

  • Neural Representation: Implicit neural representations fit individual signals with neural networks, commonly using coordinates for image, shape, and view-synthesis reconstruction.NeRV instead uses frame indices as inputs for image-wise video representation.
  • Video Compression: Traditional and learning-based video compression methods achieve reconstruction or rate-distortion benefits but can involve handcrafted components, complex pipelines, or expensive computation.Neural methods often retain principles from traditional compression while replacing parts of the pipeline.
  • Model Compression: Model compression methods applied to neural video representations include weight pruning, weight quantization, and entropy encoding for lossless compression.The paper uses only a small subset of available model-compression methods.
  • Video Inpainting & Internal Learning: Video inpainting can use external training priors or internal learning that exploits recurring information within a single image or video.The paper positions its approach closer to zero-shot, fully internal video restoration.

3. Method

HNeRV combines tiny content-adaptive frame embeddings with a learned decoder, redesigning decoder blocks to allocate more capacity to later layers. It also supports compression through model compression and embedding quantization, and downstream video inpainting through masked reconstruction loss.

  • Model Architecture: HNeRV uses a learnable encoder built from ConvNeXt blocks and novel HNeRV blocks for the decoder, while keeping frame embeddings low-dimensional.A 1 × 1 convolution produces low-dimensional frame embeddings, and a final 3 × 3 convolution produces image predictions.
  • Hybrid Neural Representation: HNeRV stores each video using tiny content-adaptive frame embeddings and a learned neural decoder, combining explicit and implicit representation approaches.The encoder generates video-specific embeddings that serve as decoder input.
  • Model Architecture: Larger kernels and wider channels in later HNeRV blocks counter NeRV’s shrinking parameter allocation, strengthening high-resolution content storage.The HNeRV block uses convolution, pixelshuffle, and activation layers, with learnable parameters only in the convolution layer.
  • Video Compression: Video compression combines global unstructured pruning, model quantization, weight entropy encoding, and embedding quantization.Entropy encoding stores sparse weights together with their locations, addressing a limitation described for NeRV compression comparisons.
  • Video Inpainting: For video inpainting, HNeRV computes reconstruction loss only on non-masked pixels and fills masked regions with its output.The mask matrix marks distorted pixels with 1 and other pixels with 0.

4. Experiments

Experiments compare HNeRV with implicit representations and codecs across video regression, parameter allocation, decoding, compression, interpolation, and inpainting. HNeRV generally improves regression quality and speed while supporting compact representation and downstream tasks, with weaker compression performance on highly dynamic scenes.

  • 4.2. Main Results: HNeRV outperforms NeRV and E-NeRV at comparable model sizes and training budgets, including 7 UVG and 4 DAVIS videos at multiple resolutions.The comparison uses 300 epochs and reports large improvements at 960×1920 and 480×960 resolutions.
  • 4.2. Main Results: 0.35M parameters and 0.003 ppp suffice for HNeRV to reconstruct video well, demonstrating compact video representation.The visualization compares HNeRV with ground truth and compares NeRV, E-NeRV, and HNeRV across five patches.
  • 4.3. Parameter Distribution Analysis: Balanced decoder parameters improve PSNR and MS-SSIM for both HNeRV and NeRV, with Kmin=1, Kmax=5, and r=1.2 maximizing both metrics.The parameter-rebalancing analysis motivates allocating more capacity to later decoder layers while avoiding vanishingly small final layers.
  • 4.4. Downstream Tasks: HNeRV outperforms H.264, H.265, and DCVC in decoding comparisons while offering flexible partial-frame decoding and favorable compactness, quality, and speed trade-offs.The decoding experiment evaluates reconstruction qualities at PSNR 32, 35, and 37 using different hardware for codec and neural methods.
  • 4.4. Downstream Tasks: HNeRV improves compression over NeRV and H.264/H.265, achieves 10× smaller size than H.265 at equivalent PSNR on a static-camera video, and remains limited on highly dynamic scenes.The compression setup applies pruning, 8-bit embedding and model quantization, and entropy encoding; finding architectures for dynamic videos is left for future work.
  • 4.4. Downstream Tasks: Content-adaptive embeddings improve held-out-frame interpolation, while HNeRV also improves inpainting over NeRV and reaches comparable performance to IIVI without task-specific design.Interpolation uses every other frame as a test set, and inpainting evaluates fixed and object masks.

5. Conclusion

The conclusion presents HNeRV as a hybrid representation that combines content-adaptive embeddings with evenly distributed decoder parameters. It reports improved regression and practical decoding, compression, and inpainting results, while acknowledging training cost, architecture-selection difficulty, and a speed trade-off.

  • 5. Conclusion: HNeRV combines content-adaptive embeddings and evenly distributed parameters to improve reconstruction quality, convergence speed, and internal generalization over implicit methods.The conclusion summarizes the method's central design and its three reported regression benefits.
  • 5. Conclusion: HNeRV is simple, fast, and flexible for video decoding and performs well for video compression and inpainting.These downstream claims are stated as properties of HNeRV as a video representation.
  • 5. Conclusion: HNeRV must be retrained for each new video, and selecting the best embedding size, model size, and architecture remains open.Later-layer widening improves regression but slightly slows the network.

A. Ablation study

The ablations vary decoder architecture, embeddings, and decoding/compression procedures to evaluate HNeRV design choices and efficiency.

  • Architecture ablations: Increasing later-layer capacity through kernel size and channel-width changes is evaluated with dedicated architecture ablations.The experiments include kernel-size and channel-reduction studies.
  • Embedding ablations: Embedding spatial size and dimension are separately varied to measure their effects on HNeRV representations.
  • Decoding evaluation: HNeRV decoding speed is quantitatively evaluated against H.264 and H.265 using decoding-FPS and decoding-time tables.The evaluation uses benchmark commands for traditional codecs and reports results in Tables 13–15.
  • Compression procedure: The video-compression pipeline combines global unstructured pruning, quantization, and entropy encoding to reduce representation size.Pruned parameters are set to zero, model and embeddings are quantized, and Huffman coding losslessly compresses quantized weights.

D. Weight Pruning for Model Compression.

The model-compression procedure addresses sparse-weight storage by combining pruning, quantization, and entropy encoding, with corrected results reported for varying sparsity.

  • Entropy encoding enables fewer bits for increasingly pruned sparse weights because all pruned weights share the frequent symbol 0.The corrected results compare quantization-only models with models that additionally apply entropy encoding.

E. HNeRV architecture details

The architecture details specify HNeRV model sizes and the main encoder, embedding, and decoder-width parameters across tasks and datasets.

  • HNeRV architecture specifications report total size, strides, encoder dimension, embedding dimension, decoder input width, channel reduction, and kernel-size bounds.These settings are organized across models used for different tasks and datasets.

F. Per-video compression results

Per-video compression performance is reported for UVG videos using HNeRV.

  • HNeRV video-compression results are reported on UVG videos in Figure 10.

G. More visualizations

This section presents additional visualizations for video regression, interpolation, inpainting, and compression results.

  • Additional visualizations cover video regression, video interpolation, and video inpainting.
  • Figure 10 presents compression results averaged across all UVG videos and for each specific video.
  • Figure 11 compares ground truth, NeRV output, and HNeRV output for video regression.
  • Figure 12 presents video interpolation results.
  • Figure 13 presents video inpainting results.
Loading 2304.02633v1…