Source-linked AI summary
DiffVC-ONE: Diffusion-based Generative Video Compression with One-Step Video Diffusion Transformer
Wenzhuo Ma, Zhenzhong Chen
TL;DR
Generative video compression must recover rich details while maintaining temporal consistency and low inference cost. DiffVC-ONE addresses this with unified latent compression, one-step Video DiT enhancement, and hybrid guidance, achieving state-of-the-art perceptual quality and temporal consistency with low inference cost.
Problem
Generative video compression needs to recover rich visual details at low bitrates while maintaining temporal consistency and low inference cost.
Method
DiffVC-ONE combines shared U2LC latent compression, one-step GOP-level OSDiT enhancement, and structural, strength, and semantic HCG guidance.
Results
DiffVC-ONE achieves state-of-the-art perceptual quality and temporal consistency with low inference cost across multiple standard benchmarks.
Takeaways & Limitations
The framework jointly performs temporal compression, condition modeling, and generative enhancement in compact video latent space while maintaining cross-frame consistency.
Abstract
from arXiv · showhide
Generative video compression can recover rich visual details at low bitrates, but simultaneously achieving high temporal consistency and low inference cost remains challenging. To address this issue, we propose DiffVC-ONE, a diffusion-based generative video compression framework built on a one-step Video Diffusion Transformer. First, we introduce a Unified Unidirectional Latent Compressor that uses a shared model to efficiently and uniformly compress compact latent slices. We then develop a Video DiT-based One-Step Diffusion Enhancer that uses the reconstructed latent slices as content anchors and performs single-step spatio-temporal perceptual enhancement over an entire group of pictures. Finally, a Hybrid Condition Generator extracts structural, strength, and semantic conditions from the reconstructed content and quantization information. These conditions preserve faithful regions, control the degree of generative enhancement, and supplement content-aware perceptual details during one-step diffusion enhancement. Extensive experiments on multiple standard benchmarks demonstrate that DiffVC-ONE achieves state-of-the-art perceptual quality and temporal consistency with low inference cost.
I. INTRODUCTION
DiffVC-ONE targets the tension among perceptual quality, temporal consistency, and inference efficiency in generative video compression. It combines unified latent compression, one-step video diffusion enhancement, and hybrid conditional guidance to address separate coding architectures, costly or inconsistent reconstruction, and insufficient conditioning.
- Motivation: Existing diffusion-based video compression methods face separate coding architectures, temporal-consistency and decoding-efficiency trade-offs, and insufficient conditional guidance.Separate intra/inter codecs can create temporal discontinuities; frame-wise diffusion may flicker, while video diffusion often requires costly multi-step sampling.
- Proposed Framework: DiffVC-ONE performs unified temporal compression, one-step video generative enhancement, and hybrid condition modeling in compact latent space.The framework is built upon a one-step Video Diffusion Transformer.
- Unified Unidirectional Latent Compressor: U2LC uses one shared coding architecture for all latent slices within a GOP, avoiding quality inconsistency, parameter redundancy, and training overhead from separate intra-frame coding.Its unidirectional reference structure is adopted as the default because experiments show the best overall trade-off among compression performance, complexity, and flexibility.
- Video DiT-based One-Step Diffusion Enhancer: OSDiT performs one-step conditional latent enhancement over an entire GOP, jointly restoring spatiotemporal textures while reducing generative-enhancement computation.Reconstructed latents act as content anchors, so diffusion recovers compression losses rather than generating from pure noise.
- Hybrid Condition Generator: HCG extracts structural, strength, and semantic conditions from reconstructed latents and quantization parameters to guide faithful, bitrate-adaptive, content-aware enhancement.These conditions preserve structure, control enhancement intensity, and guide perceptual detail recovery while reducing implausible hallucinations.
II. RELATED WORK
Related work spans neural and generative video compression, while DiffVC-ONE organizes compression and enhancement around a unified latent-space video pipeline. Its framework combines a 3D VAE, U2LC, OSDiT, and HCG to produce perceptually improved and temporally consistent reconstructions.
- Neural Video Compression: Neural video compression learns compact video representations through end-to-end optimization, advancing motion modeling, context modeling, variable-rate coding, and computational efficiency.The related-work progression includes DVC, DCVC, and subsequent DCVC variants.
- Generative Video Compression: Generative video compression includes GAN-based and diffusion-based approaches, with diffusion decoders restoring details lost during compression.GAN-based methods can suffer unstable training and unnatural artifacts, while diffusion methods incorporate pretrained diffusion models into decoders.
- DiffVC-ONE Framework: DiffVC-ONE comprises a pretrained 3D VAE, U2LC, OSDiT, and HCG for latent-space compression, conditional enhancement, and video reconstruction.The 3D VAE encodes a GOP into compact latent slices; U2LC reconstructs them, HCG derives conditions, and OSDiT enhances the latents before decoding.
A. Unified Unidirectional Latent Compressor
U2LC unifies intra- and inter-slice latent coding in a shared unidirectional architecture, using temporal, global, and spatial priors for compact latent compression.
- U2LC uses one shared architecture to encode intra- and inter-slice latents, avoiding heterogeneous reconstruction, parameter redundancy, and training complexity.
- The compressor operates on compact latent representations and caches each decoded contextual latent for subsequent slices before latent reconstruction.
- U2LC jointly uses global, temporal, and spatial priors for entropy coding, with preceding reconstructions serving as temporal context for inter slices.
- Unidirectional, joint, and bidirectional reference structures are evaluated within the latent compressor.
- The joint variant incurs high computational cost and may create a representational bottleneck, while bidirectional references struggle with large-motion cross-layer dependencies.
B. Video DiT-based One-Step Diffusion Enhancer
OSDiT performs one-step conditional enhancement on reconstructed latent slices jointly across an entire GOP, using a Video DiT prior to restore spatiotemporal details.
- Processing an entire GOP with a Video DiT exploits temporally coherent priors and targets compression losses rather than generating from pure noise.
- OSDiT jointly enhances reconstructed latent slices in a single step to produce perceptually improved video latents.
- Structural and semantic conditions guide enhancement, while the strength condition determines the sampling coefficient and diffusion timestep.
C. Hybrid Condition Generator
HCG generates structural, semantic, and strength conditions from reconstructed latents and quantization information to guide one-step diffusion enhancement.
- HCG produces complementary structural, strength, and semantic conditions because single-condition guidance may be insufficient across bitrates and video content.
- Quantization-aware feature modulation embeds qp into channel-wise scale and bias, making shared spatiotemporal features aware of bitrate constraints.
- The structural condition combines a projection of reconstructed latents with a learned residual to preserve spatial and temporal structure while correcting local distortions.
- The structural condition provides OSDiT with a fidelity constraint for one-step enhancement.
- The semantic condition injects content-aware semantics through low-rank residual modulation while retaining the prior of a constant prompt.
- HCG maps the predicted strength condition to sampling strength and timestep, applying conservative enhancement to reliable reconstructions and stronger compensation when information loss is greater.
D. Training Strategy
DiffVC-ONE is trained in three stages, progressing from latent-space compression to diffusion enhancement and then joint pixel-domain optimization with perceptual terms.
- D. Training Strategy: U2LC is first trained in latent space with a rate-distortion objective to establish basic compression capability.The objective uses rate cost and latent reconstruction distortion, with λ balancing the two terms.
- D. Training Strategy: OSDiT and HCG are then optimized in latent space to adapt the pre-trained Video DiT and learn complementary guidance conditions.
- D. Training Strategy: Finally, U2LC, HCG, and OSDiT are jointly optimized in the pixel domain with perceptual quality terms including LPIPS and DISTS.The final stage balances the quality terms using λ1, λ2, and λ3.
A. Experimental Setup
The experiments train DiffVC-ONE on OpenVid-HD and evaluate it across HEVC, UVG, and MCL-JCV using bitrate, distortion, perception, and temporal-consistency metrics.
- A. Experimental Setup: Training uses 36,971 OpenVid-HD clips, three training stages, and 256 × 256 cropped patches.The stages run for 120K, 60K, and 108K steps with batch sizes of 24, 6, and 2.
- A. Experimental Setup: Evaluation covers HEVC Classes B–E, UVG, and MCL-JCV, testing the first 96 RGB frames of each sequence.
- A. Experimental Setup: Reported metrics span BPP, PSNR, MS-SSIM, LPIPS, DISTS, FID, KID, Ewarp, FloLPIPS, tOF, and CLIP-F.
- A. Experimental Setup: Comparisons include traditional, distortion-oriented, GAN-based perceptual, and diffusion-based video compression methods.The diffusion-based baselines include DiffVC, DiffVC-OSD, GNVC-VD, and YODA.
B. Main Results
DiffVC-ONE achieves strong perceptual and temporal-consistency results across multiple datasets while reducing computation through GOP-level one-step diffusion, with most complexity concentrated in OSDiT and the VAE.
- B. Main Results: DiffVC-ONE achieves the best results on nearly all perceptual metrics and outperforms other generative methods on distortion metrics.It remains behind distortion-oriented codecs on distortion metrics, consistent with the rate–distortion–perception trade-off.
- B. Main Results: DiffVC-ONE consistently ranks among the best methods in FloLPIPS and tOF and outperforms other generative codecs in Ewarp and CLIP-F.
- B. Main Results: DiffVC-ONE achieves more stable results than GLC-video, DiffVC, and DiffVC-OSD by enhancing multiple frames in one step.Those methods exhibit noticeable temporal flickering in the visual comparison.
- B. Main Results: GOP-level one-step diffusion reduces computation per pixel and improves decoding speed despite DiffVC-ONE having more parameters.U2LC and HCG remain lightweight, while most computation is concentrated in OSDiT and the VAE.
C. Ablation Studies
Ablations show that unified latent compression improves over a separate compressor, while the unidirectional reference structure offers the best balance of performance, complexity, and flexibility.
- C. Ablation Studies: Replacing the separate latent compressor with U2LC improves overall performance while reducing unnecessary parameter redundancy.
- C. Ablation Studies: The bidirectional latent-compressor variant has an average BD-Rate of 10.83% and performs worse than the unidirectional variant.
- C. Ablation Studies: The joint variant uses a costly 3D architecture and has higher complexity with clearly inferior performance to the unidirectional variant.
- C. Ablation Studies: The unidirectional variant balances compression performance and complexity and generalizes more reliably from short training clips to longer test sequences.Joint and bidirectional variants are more sensitive to GOP-length mismatches, so the unidirectional variant is adopted by default.
2) Video DiT-based One-Step Diffusion Enhancer:
The one-step Video DiT enhancer replaces costly multi-step sampling with single-step spatio-temporal enhancement, while complementary conditions improve compression performance. Ablations identify structural, strength, and semantic guidance, with LoRA rank 32 selected as the default.
- OSDiT replaces multi-step Video DiT sampling, enabling joint pixel-domain optimization and substantially improving compression performance.The one-step design also avoids the long backpropagation path associated with multi-step sampling.
- Hybrid Condition Generator: Structural conditions improve perceptual and distortion metrics by providing spatial layouts, object boundaries, and motion cues.
- Hybrid Condition Generator: 15.26% average gain follows adding strength conditions that adapt enhancement intensity to video content and compression level.
- Hybrid Condition Generator: 20.19% KID gain follows adding semantic conditions, completing DiffVC-ONE and improving overall compression performance.
- 3.88% average improvement results from increasing LoRA rank from 16 to 32, while rank 64 adds only 0.04%; rank 32 is used by default.
- DiffVC-ONE enhances an entire GOP in one step while maintaining cross-frame consistency and reducing sampling cost.