Source-linked AI summary
Generative Neural Video Compression via Video Diffusion Prior
Qi Mao, Hao Cheng, Tinghan Yang, Libiao Jin, Siwei Ma
TL;DR
Ultra-low-bitrate neural video codecs struggle with perceptual realism and temporal coherence because image-based generative priors remain frame-level, causing flickering. GNVC-VD uses a video diffusion transformer for sequence-level latent compression and refinement, and experiments show improved perceptual quality with reduced flickering versus traditional, learned, and prior generative codecs.
Problem
At ultra-low bitrates, distortion-driven neural video compression oversmooths textures, while image-based generative priors lack temporal modeling and produce perceptual flickering.
Method
GNVC-VD unifies spatio-temporal latent compression with flow-matching sequence-level refinement guided by a pre-trained video diffusion transformer and compression-aware conditioning.
Results
GNVC-VD consistently outperforms traditional and learned codecs in perceptual quality while reducing flickering, preserving realistic motion and fine details below 0.03 bpp.
Takeaways & Limitations
Video-native diffusion priors can support perceptually coherent neural video reconstruction under extreme bitrate constraints.
Takeaways & Limitations
The refinement formulation treats e as the quantization error.
Abstract
from arXiv · showhide
We present GNVC-VD, the first DiT-based generative neural video compression framework built upon an advanced video generation foundation model, where spatio-temporal latent compression and sequence-level generative refinement are unified within a single codec. Existing perceptual codecs primarily rely on pre-trained image generative priors to restore high-frequency details, but their frame-wise nature lacks temporal modeling and inevitably leads to perceptual flickering. To address this, GNVC-VD introduces a unified flow-matching latent refinement module that leverages a video diffusion transformer to jointly enhance intra- and inter-frame latents through sequence-level denoising, ensuring consistent spatio-temporal details. Instead of denoising from pure Gaussian noise as in video generation, GNVC-VD initializes refinement from decoded spatio-temporal latents and learns a correction term that adapts the diffusion prior to compression-induced degradation. A conditioning adaptor further injects compression-aware cues into intermediate DiT layers, enabling effective artifact removal while maintaining temporal coherence under extreme bitrate constraints. Extensive experiments show that GNVC-VD surpasses both traditional and learned codecs in perceptual quality and significantly reduces the flickering artifacts that persist in prior generative approaches, even below 0.01 bpp, highlighting the promise of integrating video-native generative priors into neural codecs for next-generation perceptual video compression.
1. Introduction
Ultra-low-bitrate neural video compression improves rate–distortion performance but loses perceptual detail and temporal coherence. GNVC-VD addresses this gap with video-native, sequence-level generative refinement and reports stronger perceptual quality with less flickering.
- At ultra-low bitrates, distortion-oriented neural codecs oversmooth textures and erase fine structures, making perceptual realism an open challenge.
- Image-based generative priors improve detail recovery but remain frame-wise, lacking temporal modeling and causing perceptual flickering in video.
- GNVC-VD introduces the first generative NVC framework built on a pre-trained video diffusion transformer for sequence-level compression and refinement.
- Its codec combines contextual spatio-temporal latent compression with flow-matching denoising across intra- and inter-frame latents.
- GNVC-VD adapts decoded compressed latents rather than pure Gaussian noise, using compression-aware conditioning and two-stage training for coherent refinement.
- Extensive experiments report state-of-the-art perceptual performance below 0.03 bpp, with sharper textures and significantly less flickering than traditional, learned, and prior generative codecs.
2. Related Works
Neural video compression has strong distortion-oriented performance but loses texture at extreme compression, while image-prior generative codecs remain temporally inconsistent. GNVC-VD instead jointly encodes and refines spatio-temporal latents with a video generative prior.
- Figure 2 contrasts texture loss and unstable structures in traditional, learned, and GLC-Video reconstructions with GNVC-VD’s clean textures and stable motion.
- Existing neural video codecs achieve strong rate–distortion results but become overly smooth and lose fine textures at extremely low bitrates.
- Generative video methods such as GLC-Video and DiffVC adapt image-domain priors through frame-wise enhancement, but lack explicit temporal modeling and produce flickering.
- GNVC-VD introduces a video generative prior into neural video compression, jointly encoding spatio-temporal latents and performing unified latent refinement.
- Video diffusion transformers represent videos as latent-token sequences, supporting long-range temporal reasoning and separate modeling of appearance and motion.
3. Proposed Method
GNVC-VD unifies spatio-temporal latent compression with VideoDiT-based sequence-level flow-matching refinement. Its contextual codec and compression-aware refinement process jointly reconstruct temporally coherent, perceptually faithful video under extreme compression.
- Framework overview: GNVC-VD processes entire video sequences with joint spatio-temporal latent compression and refinement, rather than frame-wise enhancement.The framework uses a pre-trained video diffusion model to refine intra- and inter-frame latents together.
- Contextual Latent Codec: The contextual latent codec encodes predictive latents using previously decoded latents, reducing temporal redundancy while preserving motion-aware sequence representations.A temporal context feature is injected into both analysis and synthesis transforms, while quantized latents are entropy-coded.
- Flow-Matching Latent Refinement: The refinement module starts from decoded compressed latents, adds partial noise, and follows a flow-matching path toward clean latents instead of traversing from pure Gaussian noise.The noise level tN controls perturbation strength and the resulting refinement-path length.
- Flow-Matching Latent Refinement: A frozen VideoDiT supplies the pretrained velocity field, while conditioning adapters estimate a correction term from compression-domain features to compensate for quantization artifacts.The adapters modulate intermediate transformer representations and align compression cues with the diffusion latent space.
- Two-Stage Training Strategies: Two-stage training first aligns refined latents with ground-truth diffusion latents and then fine-tunes the full pipeline for perceptually faithful reconstruction.The latent-level stage addresses mismatch between quantized codec latents and the diffusion manifold before pixel-level optimization.
- Quantitative comparisons: Below 0.03 bpp, GNVC-VD achieves the best reported LPIPS and DISTS perceptual quality across HEVC-B, UVG, and MCL-JCV, outperforming traditional, learned, and generative codecs.The comparison includes HEVC, VVC, DCVC-FM, DCVC-RT, and GLC-Video.
4. Experiments
GNVC-VD is evaluated on standard video datasets against traditional, learned, and generative codecs using perceptual, temporal, qualitative, and ablation analyses. It delivers sharper, more temporally coherent reconstructions, with each major component contributing to performance.
- 4.1. Experimental Setup: Experiments use Vimeo-90k for training and HEVC Class B, UVG, and MCL-JCV for evaluation against traditional, neural, and generative codecs.The evaluation includes temporal consistency and semantic continuity comparisons on HEVC-B.
- 4.2. Comparison Results: GNVC-VD reduces BD-rate by over 98% in DISTS and 56% in LPIPS versus DCVC-RT on UVG, while also improving temporal coherence over GLC-Video.Against GLC-Video, it further reduces BD-rate by 86% in DISTS and 21% in LPIPS, with higher CLIP-F and lower Ewarp.
- 4.2. Comparison Results: At extreme compression, GNVC-VD preserves sharper structures than traditional, learned, and prior generative codecs and remains stable below 0.03 bpp.DCVC-RT produces over-smoothed results, whereas GLC-Video exhibits temporal flickering and motion inconsistencies.
- 4.3. Ablation Studies: Removing flow-matching refinement causes over-smoothing, while removing Stage I or Stage II weakens detail recovery or pixel-level adaptation.These qualitative effects support the complementary roles of the refinement and two-stage training components.
- 4.3. Ablation Studies: The full model achieves the best perceptual quality while preserving motion consistency, unlike over-smoothed variants whose apparent stability results from lost detail.The ablations remove flow-matching refinement, Stage I latent alignment, or Stage II pixel adaptation.
5. Conclusion and Future Work
The paper concludes that GNVC-VD enables perceptually coherent video reconstruction at extremely low bitrates through video-native, sequence-level latent denoising. It identifies transform-coding efficiency and diffusion-refinement speed as remaining areas for improvement.
- 5. Conclusion and Future Work: GNVC-VD improves perceptual quality and reduces flickering while preserving realistic motion and fine details below 0.03 bpp.The conclusion attributes this to a pre-trained video diffusion prior and sequence-level latent denoising.
- 5. Conclusion and Future Work: Future work should improve transform-coding efficiency and accelerate diffusion-based refinement.These are explicitly identified as directions for further advances.
- 5. Conclusion and Future Work: The evaluation uses RGB inputs, with videos converted from YUV420 using BT.709 and the first 96 frames extracted from each sequence.Inputs requiring dimensions divisible by 64 are zero-padded and cropped after decoding.
- 5. Conclusion and Future Work: Traditional codec baselines use HM-16.251 and VTM-17.02 with official low-delay configurations and RGB-scored results.Both codecs operate internally in 10-bit YUV444.
B. Model Implementation Details
The contextual latent codec separately processes anchor and predictive latents using neural transform-coding networks. Its anchor path combines residual bottleneck and attention blocks with joint space–channel probability modeling, while predictive coding follows DCVC-RT.
- B. Model Implementation Details: The Contextual Latent Codec uses separate neural networks to transform-code the anchor latent and predictive latents.The anchor is l1, while predictive latents are denoted {lt}t>1.
- B. Model Implementation Details: The anchor-latent path uses ELIC-like analysis and synthesis transforms built from residual bottleneck and attention blocks.A joint space–channel context model estimates the quantized anchor-latent distribution.
- B. Model Implementation Details: The predictive-latent path follows DCVC-RT and uses cascaded DC Blocks for its transforms.This path handles P-frame latent processing.
C.1. Additional Metrics Evaluation
Additional rate–distortion curves evaluate GNVC-VD with PSNR, MS-SSIM, and LPIPS-Alex. GNVC-VD improves perceptual fidelity over perceptual and MSE-optimized codecs, while retaining a small distortion-metric gap relative to the latter.
- C.1. Additional Metrics Evaluation: GNVC-VD provides better LPIPS-Alex perceptual quality than GLC-Video and PLVC while also improving PSNR and MS-SSIM over those perceptual codecs.The curves are reported in Fig. 9 for comprehensive baseline comparison.
- C.1. Additional Metrics Evaluation: Compared with MSE-optimized codecs, GNVC-VD has a small PSNR and MS-SSIM gap but substantially superior perceptual fidelity.LPIPS-Alex is included for completeness because it is common in learned compression, while LPIPS-VGG is used in the main paper.
C.2. Additional Analysis on Ewarp and CLIP-F
GNVC-VD achieves markedly stronger temporal coherence than the image-prior-based GLC-Video, while retaining slightly lower semantic consistency than traditional and MSE-optimized codecs.
- GNVC-VD achieves markedly stronger temporal coherence than image-prior-based GLC-Video across most test sequences.The comparison uses Ewarp and CLIP-F results reported for each sequence.
- GNVC-VD attains slightly lower semantic consistency than traditional and MSE-optimized codecs.Despite this trade-off, it substantially outperforms GLC-Video on semantic consistency.
C.3. Complexity
GNVC-VD is a large video compression model whose decoding latency remains substantial, especially at 1920 × 1080 resolution.
- GNVC-VD contains 2334.5M parameters, including 2154.5M in VideoDiT, 126.9M in the 3D VAE, and 53.1M in the Contextual Latent Codec.The parameter breakdown is reported across the framework’s major modules.
- At 1920 × 1080 on a single A800 GPU, GNVC-VD encodes at 153 ms per frame and decodes at 1557 ms per frame.The corresponding encoding/decoding times are 58/386 ms at 1080 × 720 and 25/129 ms at 640 × 480.
C.4. User Study
A user study found strong preference for GNVC-VD over traditional, neural, and image-prior-based codecs in perceptual quality and temporal stability.
- The study compared GNVC-VD with VVC, DCVC-RT, DCVC-FM, and GLC-Video using randomized left–right presentation.Randomization was used to avoid positional bias, and the subjective findings matched objective evaluations.
- Over 85% of participants preferred GNVC-VD against both traditional and neural codecs, with nearly unanimous preference against GLC-Video.Participants selected the reconstruction that better matched the reference in perceptual quality and temporal stability.
- Additional visual comparisons across HEVC Class B, MCL-JCV, and UVG show higher visual fidelity at the lowest bitrate.The comparisons include ground truth, four baseline codecs, and GNVC-VD, with bitrate and LPIPS scores shown beneath reconstructions.