Source-linked AI summary
NeRV: Neural Representations for Videos
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava
TL;DR
Video representations traditionally treat videos as frame sequences, while pixel-wise implicit representations are inefficient for whole-video encoding and decoding. NeRV represents videos as neural networks mapping frame indices to RGB images, then uses this representation for compression and denoising. It improves efficiency over pixel-wise representations, matches traditional compression performance, and outperforms standard denoising methods, while requiring longer training than traditional compression encoding.
Problem
Conventional video representations use frame sequences and task-specific pipelines, while pixel-wise implicit representations require inefficient per-pixel sampling for video representation.
Method
NeRV fits a neural network that maps a frame index to the corresponding RGB image, representing the video as an image-wise implicit neural network.
Results
NeRV improves encoding speed by 25× to 70× and decoding speed by 38× to 132× over pixel-wise implicit representations, matches traditional compression performance, and outperforms standard denoising methods.
Takeaways & Limitations
NeRV makes video compression accessible through general model-compression methods and supports denoising without a specialized denoising design.
Takeaways & Limitations
Achieving comparable PSNR and MS-SSIM requires longer training than traditional video-compression encoding.
Abstract
from arXiv · showhide
We propose a novel neural representation for videos (NeRV) which encodes videos in neural networks. Unlike conventional representations that treat videos as frame sequences, we represent videos as neural networks taking frame index as input. Given a frame index, NeRV outputs the corresponding RGB image. Video encoding in NeRV is simply fitting a neural network to video frames and decoding process is a simple feedforward operation. As an image-wise implicit representation, NeRV output the whole image and shows great efficiency compared to pixel-wise implicit representation, improving the encoding speed by 25x to 70x, the decoding speed by 38x to 132x, while achieving better video quality. With such a representation, we can treat videos as neural networks, simplifying several video-related tasks. For example, conventional video compression methods are restricted by a long and complex pipeline, specifically designed for the task. In contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches (H.264, HEVC \etc). Besides compression, we demonstrate the generalization of NeRV for video denoising. The source code and pre-trained model can be found at https://github.com/haochen-rye/NeRV.git.
1 Introduction
NeRV represents a video as a neural network that maps frame indices to RGB images, replacing frame-sequence storage with a unified implicit representation. This design accelerates encoding and decoding relative to pixel-wise implicit representations and supports compression and denoising applications.
- NeRV representation: NeRV represents each video frame as fθ(t), mapping a frame index to its corresponding RGB image.Video encoding fits the network to frames, while decoding uses feedforward inference.
- NeRV representation: Unlike pixel-wise representations, NeRV outputs a whole image using an MLP plus convolutional architecture rather than one RGB value per pixel.For a T × H × W video, pixel-wise methods sample T × H × W times, whereas NeRV samples T times.
- Efficiency: 25× to 70× faster encoding and 38× to 132× faster decoding were reported for NeRV versus pixel-wise implicit representations, with better video quality.The comparison concerns encoding speed, decoding speed, and video quality.
- Applications: NeRV reformulates video compression as model compression, enabling standard neural-network compression methods and comparable performance to H.264 and HEVC.This avoids the long, task-specific pipeline used by conventional video compression frameworks.
- Applications: NeRV also produces promising denoising results, outperforming traditional hand-crafted and ConvNets-based denoising methods without special denoising design.The paper reports high-quality denoised output from noisy video input without additional operations.
2 Related Work
Prior implicit neural representations typically map spatial or spatio-temporal coordinates to individual signal values, while video compression relies on engineered transform-coding pipelines. NeRV instead uses an image-wise representation and connects video compression to model compression.
- Implicit Neural Representation: Implicit neural representations encode signals as neural-network functions mapping coordinates to corresponding values such as RGB or volume density.They have been applied to 3D shapes, scenes, and appearance, as well as image compression.
- Video Compression: Traditional video compression methods such as MPEG, H.264, and HEVC generally rely on engineered transform coding, including DCT or wavelet transforms.These methods build on established image-compression techniques.
- Model Compression: Model compression reduces parameters in an original model while maintaining accuracy through methods including pruning, quantization, factorization, compact filters, and distillation.NeRV reformulates video compression as model compression to leverage these techniques.
- Implicit Neural Representation: Pixel-wise implicit video representations output one pixel RGB value at a time, whereas NeRV outputs an entire image from a frame index.NeRV uses an MLP plus ConvNets, contrasting with the simple MLP commonly used for pixel-wise outputs.
3 Neural Representations for Videos
NeRV represents videos as neural networks that map frame indices to RGB images, using an image-wise architecture and a loss combining pixel and structural fidelity. For compression, the fitted network is pruned, quantized, and entropy-encoded.
- Neural Representations for Videos: NeRV maps each frame index to its corresponding RGB image through a fitted neural network.The video is encoded by fitting fθ to frames, while decoding uses the network's forward mapping.
- Input Embedding: Positional encoding maps timestamps into a higher-dimensional space to help fit high-frequency video variations.The embedded timestamp is then passed to the NeRV network.
- Network Architecture: NeRV combines MLP layers with convolutional blocks so spatial locations can share convolutional kernels while producing a whole image.Directly outputting all pixels with MLPs would require too many parameters at high resolutions.
- Loss Objective: The training objective combines L1 reconstruction loss with SSIM over predicted and ground-truth frames.α controls the balance between the two loss components.
- Model Compression: NeRV compression sequentially applies video overfitting, global unstructured pruning, post-hoc quantization, and Huffman coding.Pruning zeros weights below a magnitude threshold, while entropy encoding further reduces model size by around 10%.
4 Experiments
Experiments evaluate NeRV on implicit-representation efficiency, video compression, and denoising, using standard datasets, metrics, and ablations. NeRV achieves large speedups over pixel-wise representations, comparable compression performance, and strong denoising results without specialized denoising design.
- Datasets and implementation: Experiments use Big Buck Bunny for implicit-representation comparisons and UVG for video compression, evaluating PSNR, MS-SSIM, and BPP.Big Buck Bunny contains 132 frames at 720 × 1080 resolution; UVG contains 7 videos and 3900 frames at 1920 × 1080 in total.
- Implicit representation comparison: 25× to 70× faster encoding and 38× to 132× faster decoding are reported for NeRV versus pixel-wise implicit representations, with better decoding quality.The comparison uses SIREN and NeRF baselines on Big Buck Bunny, with NeRV-S, NeRV-M, and NeRV-L model sizes.
- Compression ablation: 40% sparsity retains comparable performance with the full model, while an 8-bit model preserves video quality relative to the original 32-bit model.The compression pipeline combines video overfit, pruning, quantization, and weight encoding, with each step contributing consistently to the final results.
- Video compression: On UVG, NeRV generally achieves comparable performance to traditional and learning-based video compression methods, matching the state-of-the-art method at small BPP.At larger BPP, the reported performance gap is attributed to incomplete training caused by GPU resource limitations; longer training continues to improve quality.
- Video denoising: NeRV outperforms or matches median filtering in most denoising cases and exceeds conventional denoising methods without additional denoising design.It is also reported to outperform DIP qualitatively and quantitatively, with denoising attributed to both architecture prior and shared video data statistics.
- Ablation studies: Positional encoding substantially improves the baseline, while removing input embedding prevents the model from learning high-frequency information and lowers performance.Additional ablations examine upscale layers, normalization layers, activation functions, and the loss objective.
5 Discussion
NeRV represents videos as neural networks and applies general model compression to match traditional video compression performance. It also outperforms standard denoising methods, while requiring longer training than traditional encoding.
- NeRV represents videos by training a neural network on frame indices to output corresponding RGB images.
- General model compression lets NeRV match traditional video compression approaches without designing a long, task-specific pipeline.
- NeRV outperforms standard denoising methods in the video denoising task.
- NeRV training takes longer than traditional video compression encoding to achieve comparable PSNR and MS-SSIM.
- Future work includes improving neural architecture design and applying more advanced model compression methods for higher compression ratios.
A.1 NeRV Architecture
For 1920 × 1080 videos, NeRV uses positional encoding, a 2-layer MLP, and five upscaling blocks. Model sizes vary by changing C1 and C2.
- For 1920 × 1080 videos, NeRV applies a 2-layer MLP after positional encoding and then stacks five NeRV blocks.The blocks use upscale factors 5, 3, 2, 2, and 2.
- Different NeRV model sizes are obtained by changing C1 and C2 across specified channel configurations.The listed configurations range from (48,384) to (256,2048).
- Table 11 documents the NeRV architecture for 1920 × 1080 videos and identifies C1 and C2 as size controls.
A.2 Results on MCL-JCL dataset
The appendix reports NeRV video-compression results on the MCL-JCL dataset using rate-distortion plots.
- NeRV video-compression results on the MCL-JCL dataset are provided in Figure 11a and Figure 11b.
- Figure 11 presents rate-distortion plots for the MCL-JCV dataset.
A.3 Implementation Details of Baselines
The baseline implementation uses ffmpeg to evaluate H.264 and HEVC, extracting frames and compressing videos under medium settings with specified codec commands.
- ffmpeg produces the evaluation metrics for H.264 and HEVC baselines.
- Original and compressed YUV videos are converted into extracted frames for metric calculation.
- H.264 and HEVC videos are compressed under medium settings using codec-specific ffmpeg commands.
- The baseline commands use CRF as the Constant Rate Factor and EXT as the video container format extension.
A.4 Video Temporal Interpolation
NeRV can predict unseen interpolated video frames after training on a subset of frames, producing reasonable visual quality for videos with small motion.
- NeRV predicts unseen interpolated frames after training on a subset of frames from one video.The model receives an unseen interpolated frame index and infers the corresponding frame.
- The predicted frames have good and comparable visual quality to adjacent seen frames.
- Figure 12 reports temporal interpolation results for videos with small motion.
A.5 More Visualizations
Additional visualizations compare NeRV with H.265 compression and describe broader implications of representing videos as neural functions, including potential efficiency benefits and misuse risks.
- At similar BPP, NeRV outperforms H.265 on the Bosphorus video and produces fewer artifacts with smoother outputs.The comparison uses a smaller NeRV model and qualitative zoomed regions.
- NeRV represents videos as time functions parameterized by neural networks and may support video compression, denoising, and other tasks.
- The approach could potentially save bandwidth and accelerate media streaming, but may also be used for purposes beyond the authors’ control.