Source-linked AI summary
Video Compression with Graph-inspired Neural Representation
Changqi Wang, Ge Gao, Fan Zhang, Yue Li, Kai Zhang, Li Zhang, David Bull
TL;DR
INR video codecs learn temporal correlation implicitly, and its sufficiency for exploiting redundancy remains unclear. G-NeRV uses graph-inspired message passing with an adaptive gate and memory bank to share information between neighboring embeddings. On UVG, it reports BD-rate coding gains over NVRC and VTM 20.0 (RA).
Problem
INR codecs learn temporal correlation implicitly, leaving unclear whether this sufficiently exploits temporal redundancy compared with explicit methods.
Source: Introduction · pp. 1–2Method
G-NeRV uses message passing over temporal neighborhoods of frame embeddings, with an adaptive gate controlling neighbor-information sharing.
Source: Introduction · p. 2Results
8.86% and 14.68% coding gains over NVRC and VTM 20.0 (RA), respectively, are reported in BD-rate on UVG.
Source: Introduction · p. 2Takeaways & Limitations
A memory bank stores neighboring embeddings to support neighborhood construction under random frame-index sampling in INR codecs.
Source: Introduction · p. 2G-NeRV requires seeing an entire video sequence or many frames during encoding to maintain coding performance comparable to conventional codecs.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Implicit Neural Representations (INR) provide a compact and content-adaptive paradigm for video compression, typically representing a video through shared network parameters and frame-indexed embeddings. Compared to conventional or autoencoder-based codecs, these approaches exploit temporal redundancy within videos in an implicit manner, which potentially results in sub-optimal compression performance. In this paper, we propose G-NeRV, a graph-inspired INR that explicitly improves temporal redundancy exploitation in the implicit latent space. Motivated by the total correlation principles in information theory, we construct a temporal neighborhood over frame embeddings and perform message passing to aggregate reusable information from neighboring frames through an adaptive gate controlling the injection of neighboring information. Inspired by the reference frame buffer in conventional video coding, a memory bank mechanism has been further designed to enable efficient temporal-neighbor retrieval under random frame-index sampling in INR training. This new representation model has been integrated into an advanced representation compression framework and compared with existing conventional and neural video codecs. The results show that the G-NeRV codec outperforms the state-of-the-art INR-based codec, NVRC, and the latest standard video codec, VVC VTM, by 8.86\% and 14.68\% (in BD-rate), respectively, measured by PSNR on the UVG dataset.
Introduction
INR codecs offer content-adaptive video representations, but how well their implicit temporal modeling exploits redundancy remains unclear. G-NeRV introduces graph-inspired message passing and a memory bank, with reported coding gains over existing codecs on UVG.
Introduction
- Conventional codecs use a hybrid pipeline, while neural codecs replace hand-designed modules with end-to-end optimized networks; fixed inference parameters can make neural codecs sensitive to distribution shifts and costly to decode.The stated sensitivity and decoding-complexity concerns motivate content-adaptive INR approaches.Source: Introduction · p. 1
- INR codecs overfit lightweight networks to individual videos, but their temporal correlations are learned implicitly and their sufficiency relative to explicit methods is unclear.The paper identifies temporal redundancy exploitation as an area not fully investigated for INR codecs.Source: Introduction · pp. 1–2
- G-NeRV aggregates information from neighboring frame embeddings through message passing, using an adaptive gate to control sharing and a memory bank to retrieve temporal neighbors during random-index training.The figure contrasts the message-passing mechanism and memory bank with most existing INR frameworks.Source: Figure 1 · p. 1, Introduction · p. 2
- 8.86%, 14.68%, and 32.73% coding gains over NVRC, VTM 20.0 (RA), and DCVC-RT, respectively, are reported on UVG.G-NeRV also outperforms DNeRV by 7.91% in BD-rate.Source: Introduction · p. 2
Related Work
Related work contrasts pretrained neural codecs with sequence-adaptive INR representations, surveys differing approaches to temporal redundancy in INR codecs, and notes that graph-inspired message passing has rarely been applied to INR video embeddings.
Neural Video Compression
- Neural video codecs jointly optimize transforms, prediction, and entropy models, while INR codecs use overfitted representations; NeRV conditions shared decoder parameters on frame or patch indices.COOL-CHIC-video instead uses lightweight learnable latents and a compact synthesis network.Source: Neural Video Compression · p. 2
Temporal Redundancy Exploitation
- COOL-CHIC-based methods use explicit inter-coding, whereas NeRV latent embeddings do not directly support pixel-domain prediction or warping; other NeRV methods model temporal redundancy indirectly.These indirect approaches include motion or frame differences, adaptive sampling, and temporal modeling biases; the passage reports that they remain behind leading INR and conventional codecs.Source: Temporal Redundancy Exploitation · p. 2
Graph Message Passing
- Graph message passing aggregates information from neighboring nodes, but its use for temporal redundancy exploitation in INR video embeddings has rarely been explored.Prior graph-inspired applications cited here concern low-level vision tasks and spatial neighbors.Source: Graph Message Passing · pp. 2–3
Methodology
G-NeRV explicitly models temporal redundancy among frame-indexed embeddings using local neighborhoods and graph-inspired message passing. Adaptive gates control neighbor information, while a memory bank supports retrieval during training and the representation is integrated into a compression codec.
Preliminaries
- The method separates reusable, predictable embedding content from frame-specific motion, occlusion, and deformation, seeking greater effective temporal correlation without indiscriminate coupling.Greater effective temporal correlation can reduce the description length of reusable components, while excessive coupling may oversmooth motion.Source: Preliminaries · p. 3
Overall Framework
- The framework adds graph-inspired message passing after grid encoding and each decoding block, with a memory bank providing local temporal neighbors to decoder blocks.The final decoder output passes through an output head to reconstruct the frame.Source: Overall Framework · p. 3
Temporal Message Passing Mechanism
- Multiscale temporal convolutions produce center-frame messages that are mixed, then spatial- and channel-wise gates regulate their injection to preserve frame-specific details.Different temporal receptive fields capture short-range and longer-range dependencies; a residual connection preserves the original representation.Source: Temporal Message Passing Mechanism · p. 4
Memory Bank
- The memory bank stores down-scaled embeddings for neighbor retrieval under random frame-index sampling; index-order decoding and an auxiliary queue avoid repeated computation.Retrieved low-resolution neighbors are restored to full resolution for message passing.Source: Memory Bank · pp. 4–5
G-NeRV Video Codec
- G-NeRV is integrated into NVRC with learnable quantization and entropy coding, optimized end-to-end by a rate-distortion objective; only the memory bank’s D and U parameters are transmitted.The rate includes quantized and entropy-coded INR, quantization-model, and entropy-model parameters.Source: G-NeRV Video Codec · p. 5
Experiments
Experiments compare G-NeRV with codec families across three video datasets and test its components and neighborhood size. The reported UVG results show gains over VTM 20.0 (RA), DCVC-FM, and NVRC.
Implementation Details
- G-NeRV uses four complexity scales, with neighborhood radius K = 3 and channel and spatial downscale ratios of 2 and 4 in memory-bank construction.The downscaling is applied in the memory-bank module.Source: Implementation Details · p. 5
Databases and Metrics
- Evaluation covers UVG, MCL-JCV, and HEVC Class B using PSNR and MS-SSIM, with BD-rate, encoding and decoding FPS, kMACs/px, and model parameters reported.PSNR uses weighted Y-, U-, and V-channel values; MS-SSIM uses the Y-channel, and FPS is measured on the same GPU.Source: Databases and Metrics · p. 5
Baseline Methods
- Baselines span standardized, autoencoder-based, and INR-based codecs; Figure 3 presents rate-distortion curves for UVG, MCL-JCV, and HEVC B.The comparison includes HEVC HM 18.0, VVC VTM 20.0, DCVC-FM, DCVC-RT, HiNeRV, and NVRC.Source: Baseline Methods · pp. 5–6, Figure 3 · p. 6
Overall Performance
- 14.68%, 25.28%, and 8.86% BD-rate gains over VTM 20.0 (RA), DCVC-FM, and NVRC, respectively, are reported on UVG.Similar performance is reported on HEVC B; on MCL-JCV, G-NeRV trails VTM but exceeds INR-based codecs.Source: Overall Performance · p. 6, Table 1 · p. 7
Ablation Studies
- Ablations test the multi-scale convolution, adaptive gates, temporal message passing, decoding update strategy, and an alternative DNeRV mechanism in NVRC.The experiments use UVG PSNR, with the original model as the BD-rate anchor.Source: Ablation Studies · pp. 6–7
- Larger temporal radius K improves rate-distortion performance but increases encoding and decoding complexity; the authors empirically choose K = 3.Variants V1.1–V4.1 have inferior performance through BD-rate losses or complexity increases, and the memory-bank-free message-passing ablation was infeasible for HD encoding.Source: Ablation Studies · p. 7
Conclusion
The conclusion presents G-NeRV as a graph-inspired INR framework for explicit temporal-redundancy exploitation and reports a UVG coding gain. It also identifies encoding latency as a limitation and describes possible applications beyond video compression.
Conclusion
- 8.86% BD-rate savings measured by PSNR on UVG are reported when G-NeRV is integrated into an advanced neural representation compression framework.The framework uses gated message passing between latent embeddings and a memory bank that stores neighboring embeddings.Source: Conclusion · p. 7
- The message-passing module may be adapted to INR-based dynamic 3D scene reconstruction and compression, although this potential has not been fully tested.The paper also notes potential application to existing INR-based video codecs.Source: Conclusion · p. 7
- Encoding latency remains a limitation: maintaining coding performance comparable to conventional codecs requires seeing an entire sequence or many frames during encoding.The paper leaves reducing this latency as future work.Source: Conclusion · p. 7
Pseudo-code for G-NeRV
The encoding and decoding pseudocode uses shared notation for temporal neighborhoods, neighborhood radius, memory-bank operations, and codec parameters.
Pseudo-code for G-NeRV
- The notation h(ℓ)_t represents the temporal neighborhood centered at frame index t in grid encoding and decoding layer ℓ.K denotes the neighborhood radius.Source: Pseudo-code for G-NeRV · p. 10
- In the decoding layer, p(·) denotes the up-sampling block; M denotes the memory bank, and D its down-sampling module.The memory bank and down-sampling module are used in the memory-bank operations.Source: Pseudo-code for G-NeRV · p. 10
- The learnable parameters in the quantization and entropy models are denoted by ϕ and ψ, respectively.Source: Pseudo-code for G-NeRV · p. 10
Implementation Details
The implementation combines NVRC-shared coding settings with G-NeRV-specific latent configurations, patch-based training, and two-stage optimization. Supplementary comparisons report improvements over C3 and NVRC.
G-NeRV Configuration
- Four rate configurations differ only in latent-space dimensions, while quantization and entropy-model settings follow NVRC.Table 4 distinguishes G-NeRV-specific hyperparameters from those shared with NVRC.Source: G-NeRV Configuration · p. 10, Table 4 · p. 12
Training Details
- Training predicts 120×120 frame patches with neighborhoods from spatially corresponding patches in adjacent frames, then uses frame-wise inference to avoid patch-boundary artifacts.Each frame is divided into 144 patches; embeddings are padded during training to keep patch-wise training consistent with frame-wise decoding.Source: Training Details · p. 10
Loss
- The total loss combines ℓ2 and Y-channel MS-SSIM terms; the weighting was not thoroughly studied, though the chosen ratio gave good PSNR and MS-SSIM.The MS-SSIM term uses a window size of 5 and is aligned with PSNR-YUV (6:1:1) and MS-SSIM (Y) metrics.Source: Training Details · p. 10
Optimization
- Two-stage Adam training uses learning rates of 2×10−3 and 1×10−4; UVG and HEVC use 400 and 20 epochs, while MCL-JCV doubles both counts.The first stage uses soft rounding and Kumaraswamy noise; the second uses Quant-Noise, and R is optimized once every eight D optimization steps.Source: Training Details · pp. 10–11
More Experimental Results
- Figure 6 compares ground-truth frames with NVRC and G-NeRV reconstructions, and the paper reports superior reconstruction quality for G-NeRV.Source: More Experimental Results · p. 11, Figure 6 · p. 12
- G-NeRV improves PSNR and MS-SSIM over C3 at comparable bit rates, with higher decoding complexity; MCL-JCV is omitted because quality ranges do not overlap.Source: Figure 5 · p. 11, More Experimental Results · p. 11, Table 5 · p. 12