DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencoder

1. Introduction

• Video diffusion models’ computational demands limit their accessibility and practical deployment.The introduction situates this challenge amid advances in diffusion models and large-scale training for high-quality, temporally coherent video generation.Source: §1 · p. 2
• 14.8× inference speedup over the base model accompanies generation up to 2160×3840 on one H100; adapting Wan-2.1-14B takes 10 H100 GPU days, 230× less training cost.The reported evaluations find comparable or superior VBench scores across reconstruction, text-to-video, and image-to-video tasks.Source: §1 · p. 2
• DC-AE-V achieves 32×/64× spatial and 4× temporal compression while preserving reconstruction quality and generalization to longer videos.Its chunk-causal design combines bidirectional information flow within chunks with causal information flow across chunks.Source: §1 · p. 2
• AE-Adapt-V transfers pre-trained models to DC-AE-V’s latent space through lightweight fine-tuning, aligning the patch embedder and output head for robust initialization.LoRA fine-tuning then rapidly recovers the base model’s quality.Source: §1 · p. 2
• Across video reconstruction, text-to-video, and image-to-video, DC-VideoGen delivers efficiency gains with comparable or superior VBench scores.The evaluations are reported in Table 1 and Tables 2–4.Source: §1 · p. 2

Video Autoencoder

• Prior video autoencoders commonly use 8× spatial and 4× temporal compression, while some use 16× spatial compression; 32× approaches have faced reconstruction-quality or longer-video generalization problems.DC-AE-V is presented as achieving up to 64× spatial compression while maintaining reconstruction quality and generalization.Source: §2 · p. 3

Efficient Autoencoder Adaptation for Video Diffusion Models

• OpenSora 2.0 [25] adapted pre-trained models by loading DiT backbone weights and randomly initializing the patch embedder and output head, but produced blurry videos and did not match training-from-scratch performance.The paper introduces embedding-space alignment to recover the base model’s knowledge in the new latent space.Source: §2 · p. 3

Video Diffusion Model Acceleration

• Other video-diffusion acceleration work reduces diffusion steps or compresses models through sparsity and quantization; DC-VideoGen instead reduces token redundancy.Source: §2 · p. 3

3.1. DC-VideoGen Overview

• Applying DC-VideoGen to Wan-2.1-14B takes 10 H100 GPU days, or 0.05% of MovieGen-30B’s training cost.Source: §3.1 · p. 3

3.1.1. Preliminaries and Notation

• The fxtycz notation specifies an autoencoder configuration; at equal reconstruction quality, higher compression is preferred, and allocating more spatial compression to the autoencoder than the patch embedder yields better generation results.Source: §3.1.1 · p. 4

3.2. Deep Compression Video Autoencoder

• Chunk-causal modeling uses bidirectional information flow within fixed-size chunks and causal flow across chunks, with reconstruction quality improving as chunk size grows and benefits plateauing beyond 40.The final design uses chunk size 40 because larger chunks raise training costs after quality benefits plateau.Source: §3.2 · pp. 4–5
• DC-AE-V achieves higher reconstruction accuracy than causal autoencoders at the same compression ratio and better quality and longer-video generalization than non-causal autoencoders; DC-AE-V-f64t4c128 gives a 22× speedup with slightly higher VBench scores than the base model.The generation comparison uses Wan-2.1-T2V-1.3B adapted with the same AE-Adapt-V setup across autoencoders.Source: Table 1 · p. 5, §3.2 · p. 6, Table 2 · p. 6

3.3.1. Naïve Approach, Challenge and Analysis

• Retaining pre-trained DiT blocks while randomly initializing the patch embedder and output head fails to match the base model’s semantic score and degrades to random noise after 20K steps.The authors conjecture that the new latent space and randomly initialized patch embedder create an embedding-space gap that impedes retention of pre-trained knowledge.Source: §3.3.1 · pp. 6–7

3.3.2. Our Solution: AE-Adapt-V

• AE-Adapt-V aligns the patch embedder and output head before end-to-end LoRA fine-tuning; patch alignment is most critical, and LoRA achieves higher VBench scores and visual quality than full fine-tuning.Alignment bridges embedding spaces and preserves pre-trained knowledge; output-head alignment further enhances quality.Source: §3.3.2 · pp. 7–8

3.4. DC-VideoGen Application

• DC-VideoGen applies to pre-trained video diffusion models and is evaluated on text-to-video and image-to-video generation; for I2V, image conditions are repeated and blank frames appended to form chunks before encoding.Source: §3.4 · p. 8

4.1. Setups

• The experiments used three pretrained Wan-2.1 models adapted from the original Wan-2.1-VAE to DC-AE-V.Training data combined 257K synthetic videos with 160K high-resolution videos.Source: §4.1 · p. 8
• Models were benchmarked on a single H100 GPU using TensorRT, with latency measured for the transformer backbone.The backbone was selected because it is the primary efficiency bottleneck; T2V and I2V quality were evaluated with VBench and VBench 2.0, respectively.Source: §4.1 · p. 8

4.2. Text-to-Video Generation

• 7.7× lower latency and a VBench score increase from 83.73 to 84.83 versus Wan-2.1-14B are reported for text-to-video generation.DC-VideoGen-Wan-2.1 also has the highest VBench score and lowest latency among the compared T2V models.Source: §4.2 · p. 9, Table 3 · p. 9

4.3. Image-to-Video Generation

• 7.6× lower latency and a higher VBench score than base Wan-2.1-14B are reported for image-to-video generation.Against other I2V models, it runs 5.8× faster than MAGI-1 and 8.3× faster than HunyuanVideo-I2V.Source: §4.3 · p. 9, Table 4 · p. 9

5. Conclusion

• Up to 14.8× faster inference is achieved while preserving or improving video quality.The framework combines a deep-compression video autoencoder with an efficient adaptation strategy.Source: §5 · p. 10
• DC-VideoGen drastically reduces training costs compared with training video diffusion models from scratch.The conclusion frames efficiency and fidelity as advancing together.Source: §5 · p. 10
• The authors conclude that these findings make large-scale video synthesis more practical and accessible for research and real-world applications.Source: §5 · p. 10

A.2.1. Model Architecture

• The f32t4c32 DC-AE-V encoder and decoder each contain six stages built from 3D ResNet blocks.Source: §A.2.1 · p. 15, Figure 9 · p. 15
• The first five stages perform spatial downsampling or upsampling, while the final stage handles temporal downsampling or upsampling.Source: §A.2.1 · p. 15
• The architecture incorporates Residual Autoencoding for optimization and extends the StyleGAN2 discriminator to process video inputs.Source: §A.2.1 · p. 15

A.2.2. Dataset

• DC-AE-V is trained on a mixture of video and image datasets.Source: §A.2.2 · p. 15
• The video training data include subsets of Panda70m and OpenVid1m.Source: §A.2.2 · p. 15
• The image training data include ImageNet21k, Mapillary Vistas, DataComp, WiderFace, WiderPerson, TextCaps, and Unsplash.Source: §A.2.2 · p. 15
• Table 5 presents additional video reconstruction results.Source: Table 5 · p. 16

• Four metrics—PSNR, SSIM, LPIPS, and FVD—are used to evaluate video autoencoders on 80×256×256 videos.Source: §A.2.3 · p. 16
• The evaluation set combines 1,000 unseen Panda70m videos, 3,783 UCF101 test videos, 5,044 ActivityNet 1.3 test videos, and 5,000 Kinetics600 test videos.Source: §A.2.3 · p. 16
• When an autoencoder cannot process 80-frame videos, extra frames are padded at the end and excluded from reconstruction metrics.Source: §A.2.3 · p. 16

A.2.4. Additional Reconstruction Results

• DC-AE-V achieves superior reconstruction accuracy and longer-video generalization across benchmarks, with particular gains for small text and human faces.Source: §A.2.4 · p. 16

A.3. Ablation Study on Video Embedding Space Alignment

• Aligning both the patch embedder and output head gives the best results, while patch embedder alignment is most critical to overall performance.Source: §A.3 · p. 16, Figure 11 · p. 18

A.6. Detailed Training Hyperparameters of AE-Adapt-V

• Under deep compression, causal autoencoders have low reconstruction quality, while non-causal autoencoders reconstruct better but generalize poorly to longer videos; Table 8 lists AE-Adapt-V hyperparameters.Source: §A.6 · p. 16, Figure 10 · p. 17, Table 8 · p. 19

A.4. Detailed Evaluation Results on VBench

• DC-VideoGen-Wan-2.1-T2V-1.3B outperforms the base model on 11 of 16 VBench metrics.Source: §A.4 · p. 16, Table 6 · p. 17

A.5. Detailed Efficiency Benchmark Results

• Table 7 reports detailed efficiency results measured on an NVIDIA H100 GPU.Source: §A.5 · p. 16, Table 7 · p. 17

A.7. Qualitative Comparison with the Pre-trained Models

• DC-VideoGen-Wan2.1-I2V-14B and DC-VideoGen-Wan2.1-T2V-14B retain their base models’ generation quality while reducing latency by around 87%.Source: §A.7 · pp. 17–18, Figure 12 · p. 18, Figure 13 · p. 19

A.8. Limitation and Future Work

• Performance is strongly dependent on pre-trained-model quality, and extending the framework to long-video generation is planned future work.Source: §A.8 · p. 18