PaperlayerView exact revision on arXiv

Source-linked AI summary

DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space

Wenkun He, Yuchao Gu, Junyu Chen, Dongyun Zou, Yujun Lin, Zhekai Zhang, Haocheng Xi, Muyang Li, Ligeng Zhu, Jincheng Yu, Junsong Chen, Enze Xie, Song Han, Han Cai

arXiv:2509.25180v3cs.CVcs.AI

TL;DR

High-resolution diffusion generation is slowed by redundant latent representations, while training models from scratch in deeply compressed spaces is costly. DC-Gen adapts pretrained models through post-training and embedding alignment. It preserves pretrained-model performance across several tasks while delivering substantial speedups for high-resolution generation.

  • Problem

    Moderately compressed latent spaces retain redundancy, while training high-quality diffusion models from scratch in deeply compressed spaces is computationally and data intensive.

    Source: §1 · pp. 1–2

  • Method

    DC-Gen adapts pretrained diffusion models to deeply compressed latent spaces through post-training, using embedding alignment to bridge representation differences.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across image generation, video generation, and image editing, DC-Gen preserves pretrained-model performance while delivering substantial speedups for high-resolution generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DC-Gen demonstrates a post-training route to using deeply compressed latent spaces across image, video, and image-editing tasks.

    Source: §1 · p. 2

Abstract

from arXiv · show

Existing text-to-image diffusion models excel at generating high-quality images, but face significant efficiency challenges when scaled to high resolutions, like 4K image generation. While previous research accelerates diffusion models in various aspects, it seldom handles the inherent redundancy within the latent space. To bridge this gap, this paper introduces DC-Gen, a general framework that accelerates text-to-image diffusion models by leveraging a deeply compressed latent space. Rather than a costly training-from-scratch approach, DC-Gen uses an efficient post-training pipeline to preserve the quality of the base model. A key challenge in this paradigm is the representation gap between the base model's latent space and a deeply compressed latent space, which can lead to instability during direct fine-tuning. To overcome this, DC-Gen first bridges the representation gap with a lightweight embedding alignment training. Once the latent embeddings are aligned, only a small amount of LoRA fine-tuning is needed to unlock the base model's inherent generation quality. We verify DC-Gen's effectiveness on SANA and FLUX.1-Krea. The resulting DC-Gen-SANA and DC-Gen-FLUX models achieve quality comparable to their base models but with a significant speedup. Specifically, DC-Gen-FLUX reduces the latency of 4K image generation by 53x on the NVIDIA H100 GPU. When combined with NVFP4 SVDQuant, DC-Gen-FLUX generates a 4K image in just 3.5 seconds on a single NVIDIA 5090 GPU, achieving a total latency reduction of 138x compared to the base FLUX.1-Krea model. Code: https://github.com/dc-ai-projects/DC-Gen.

1. Introduction

High-resolution diffusion models carry redundant latent tokens, while training new models in deeply compressed spaces is costly. DC-Gen adapts pretrained models through post-training, reporting substantial speedups while preserving quality.

1. Introduction

  • High-resolution diffusion inference remains slow, and moderately compressed latent spaces retain redundancy that adds computational cost.FLUX.1-Krea represents a 4K image with 65,536 visual tokens; generating one takes over three minutes on an NVIDIA H100 GPU.Source: §1 · pp. 1–2
  • DC-Gen uses cost-efficient post-training to adapt pretrained diffusion models to deeply compressed latent spaces, where embedding alignment bridges the representation gap before further training.Training high-quality models from scratch in these spaces remains challenging because of computational and data demands.Source: §1 · p. 2
  • 53.8× speedup in 4K image generation is reported for DC-Gen-FLUX versus pre-trained FLUX.1-Krea.The post-training process took 40 H100 GPU days.Source: §1 · p. 2
  • Across image generation, video generation, and image editing, DC-Gen preserves pretrained-model performance while delivering substantial speedups for high-resolution generation.Source: §1 · p. 2

2. Related Work

Prior work explores compressed latent spaces and several diffusion-acceleration strategies. DC-Gen combines these directions by adapting pretrained models to deeply compressed spaces through post-training.

2.1. Latent Diffusion Models

  • DC-Gen leverages existing pre-trained models and deeply compressed latent spaces to achieve acceleration through efficient, stable post-training, without training a new autoencoder or introducing additional procedures.Source: §2.1 · p. 2

2.2. Diffusion Model Acceleration

  • Diffusion acceleration includes training-free methods, post-training approaches, efficient architectures, and trajectory modeling; DC-Gen is a post-training approach using deeply compressed latents.Training high-quality models from scratch in such spaces remains expensive.Source: §2.2 · p. 3

3. Method

DC-Gen moves pretrained diffusion models from 8× latent compression to 32× or 64× through post-training, avoiding the cost and data demands of training from scratch. Its pipeline aligns the new latent representation before fine-tuning, addressing instability from direct adaptation and retaining the pretrained model’s generation capability.

3.1. Motivation and Overview

  • The pipeline targets 32× or 64× spatially compressed latents instead of the original 8× space, reducing visual-token counts and supporting faster inference and downstream fine-tuning.The paper presents post-training as a way to avoid the cost and high-quality data demands of training from scratch.Source: §3.1 · p. 3

3.2. Preliminary

  • Patch embedders convert latents into transformer tokens and output layers map embeddings back to latents; both modules depend on latent channel count and patch size.Consequently, they cannot be reused unchanged when adapting to latent spaces with different shapes.Source: §3.2 · p. 3

3.3. Challenge, Analysis, and the DC-Gen Pipeline

  • Direct fine-tuning reuses the pretrained transformer but randomly initializes the new patch embedder and output layer, and can yield degraded quality or unstable, near-noise outputs.DC-Gen attributes this failure to a representation gap: random embeddings diverge from pretrained representations through transformer layers, hindering recovery of pretrained capabilities.Source: §3.3 · p. 4
  • DC-Gen aligns embeddings before end-to-end fine-tuning, spatially downsampling pretrained embeddings and optimizing the new patch embedder to reduce their distance.After patch-embedder training, it jointly tunes the patch embedder and output layer for a few steps with the transformer frozen; brief LoRA fine-tuning then refines details and restores generation quality.Source: §3.3 · pp. 4–5
  • Embedding alignment reduces per-layer feature distance and enables semantically reasonable generation in the new latent space without fine-tuning the diffusion backbone.These results are reported for FLUX.1-Krea.Source: Figure 2 · p. 4, §3.3 · p. 5

3.4. Discussion and Ablation Study on Embedding Alignment

  • Without alignment, direct fine-tuning blurs FLUX.1-Krea images, distorts Qwen-Image-Edit details and editing behavior, and can collapse Wan2.1 video generation to near-noise.The ablation compares DC-Gen with direct fine-tuning and training from scratch using DC-AE-f32 as the target latent space.Source: §3.4 · pp. 6–7
  • After alignment, DC-Gen needs only a few stable end-to-end fine-tuning steps to reach visual quality comparable to the pretrained model.The ablations also find that patch-embedder alignment is crucial for recovering pretrained capabilities, while output-layer alignment speeds convergence.Source: §3.4 · p. 7

3.5. Additional Techniques and Implementation Details

  • For guidance-distilled text-to-image models, the paper replaces the standard flow-matching objective with a corrected objective to address biased velocity estimation.For video generation, it develops DC-AE-V with chunk-causal temporal modeling: bidirectional information flow within chunks and causal flow across chunks.Source: §3.5 · p. 7

4. Experiments

Experiments evaluate DC-Gen on text-to-image, text-to-video and image-to-video, and image editing. Across these tasks, it reports quality comparable to pretrained models with reduced latency.

4. Experiments

  • Experiments cover text-to-image, text-to-video, image-to-video, and image editing, with latency benchmarked using TensorRT on a single H100 GPU.Latency measurements focus on the transformer backbone, identified as the primary efficiency bottleneck.Source: §4 · p. 8

4.1. Text-to-Image Generation

  • At 1K, DC-Gen-FLUX and DC-Gen-Z-Image achieve comparable quantitative results with about 4× lower latency; at 4K, latency is reduced by over 50×.Human evaluation also found comparable visual quality and text alignment across general, person, and anime domains.Source: Table 1 · p. 8, §4.1 · p. 8, Table 2 · p. 9

4.2. Text-to-Video and Image-to-Video Generation

  • At 720×1280, DC-Gen-Wan2.1 achieves comparable results to pretrained Wan2.1 models; the 14B text-to-video model reduces latency by 7.7×.The DC-Gen-Wan2.1 series also reports relatively high VBench scores and the lowest latency compared with other pretrained diffusion models.Source: §4.2 · pp. 9–10, Table 4 · p. 10

4.3. Image Editing

  • DC-Gen-Qwen-Image-Edit achieves comparable editing performance while reducing latency by 4.8×.The evaluation uses GEdit, which measures instruction fidelity, output-image quality, and overall editing behavior.Source: §4.3 · pp. 10–11, Table 5 · p. 11

5. Conclusion

DC-Gen adapts pre-trained diffusion models to deeply compressed latent spaces through embedding alignment and end-to-end fine-tuning. Across image generation, video generation, and image editing, the resulting models match their pre-trained counterparts’ performance while gaining efficiency.

5. Conclusion

  • DC-Gen adapts pre-trained diffusion models to deeply compressed latent spaces to accelerate them.The framework uses embedding alignment followed by end-to-end fine-tuning to address quality degradation after changing latent spaces.Source: §5 · p. 11
  • Embedding alignment followed by end-to-end fine-tuning addresses quality degradation after switching latent spaces.The alignment procedure is intended to preserve the pre-trained model’s generative capabilities.Source: §5 · p. 11
  • Across image generation, video generation, and image editing, DC-Gen models match their pre-trained counterparts’ performance while achieving substantial efficiency gains.The paper also reports that DC-Gen can be combined with other orthogonal acceleration methods.Source: §5 · p. 11

A. Corrected Flow Matching Objective for Guidance-Distilled Models

Guidance-distilled models are trained to mimic classifier-free guidance, so applying the standard flow-matching objective produces biased velocity estimates. The section derives a corrected velocity estimate from the distilled model and uses it in a corrected training objective.

Problem

  • The standard flow-matching objective gives biased velocity estimates for guidance-distilled models because those models are trained to mimic classifier-free guidance.The paper identifies this bias as a problem with directly applying the standard objective to guidance-distilled models.Source: A. Corrected Flow Matching Objective… · p. 12

Correcting the Velocity Estimation

  • The correction derives the raw velocity from the distilled model by approximating the unconditional velocity with its output under an empty condition.The derivation treats the distilled model’s output as the classifier-free-guidance output.Source: A. Corrected Flow Matching Objective… · p. 12

Corrected Training Objective

  • The corrected velocity estimate is incorporated into a revised flow-matching objective for training guidance-distilled models.The passage states that this corrected objective allows direct training of the guidance-distilled model.Source: A. Corrected Flow Matching Objective… · p. 12

B.1. Embedding Alignment

Supplementary experiments compare embedding alignment with other adaptation strategies, examine LoRA fine-tuning, and evaluate DC-Gen on image and video generation. They report improved 4K image quality and efficiency over Diffusion-4K, as well as comparable visual quality with reduced training memory costs for LoRA fine-tuning.

B.1. Embedding Alignment

  • Embedding alignment outperforms autoencoder-level latent alignment, and tested alternative adaptation strategies do not match DC-Gen’s visual quality.The alternatives include layer-wise learning-rate decay and gradually introducing LoRA modules; the comparison uses FLUX.1-Krea.Source: §B.1 · p. 13, Figure 12 · p. 14, Table 7 · p. 15

DC-Gen-FLUX

  • The patch embedder alignment and output layer alignment phases are both described as indispensable for training stability and faster end-to-end fine-tuning convergence.Source: DC-Gen-FLUX · p. 14

B.2. LoRA Fine-Tuning

  • LoRA fine-tuning reduces training memory costs by reducing trainable parameters and yields better visual quality than the compared end-to-end fine-tuning techniques.The authors suggest improved preservation of the pre-trained model’s generative capabilities might explain the visual-quality result.Source: §B.2 · p. 15, Table 8 · p. 15, Figure 13 · p. 16

C.1. Text-to-Image Generation

  • DC-Gen achieves much better visual quality and significantly higher efficiency than Diffusion-4K in the reported 4K image-generation comparison.The comparison notes that pre-trained FLUX.1-Krea does not support native 4K generation.Source: §C.1 · p. 15

C.2. Text-to-Video and Image-to-Video Generation

  • The section compares videos from DC-Gen-Wan2.1 text-to-video and image-to-video models with their corresponding pre-trained Wan2.1-14B models.Source: §C.2 · p. 16

C.3. Image Editing

The image-editing examples show DC-Gen-Qwen-Image-Edit supporting different resolution ratios in English and Chinese. The authors report image quality comparable to Qwen-Image-Edit and provide additional editing samples.

C.3. Image Editing

  • Figure 17 presents additional image-editing samples.Source: §C.3 · p. 16, Figure 17 · p. 20
  • DC-Gen-Qwen-Image-Edit achieves image quality comparable to Qwen-Image-Edit while supporting different resolution ratios in English and Chinese.Source: §C.3 · p. 16
  • The examples include edits prompted in both English and Chinese, such as changing a cat’s action and replacing a background.Source: §C.3 · p. 20

D. Human Evaluation

Human evaluations report comparable image quality and text alignment for DC-Gen-Z-Image and Z-Image-Turbo, and near-base-model results for image editing. Video comparisons show a slightly higher DC-Gen win rate, while training uses lower resolutions before resolution lifting.

D.1. Text-to-Image Generation

  • DC-Gen-Z-Image achieves visual quality and text alignment similar to pre-trained Z-Image-Turbo in human evaluation.The study covered general, anime, and real-human categories, with 25 image pairs per category and 20 users per pair.Source: §D.1 · p. 21, Table 9 · p. 21

D.2. Text-to-Video and Image-to-Video Generation

  • DC-Gen-Wan2.1 receives a slightly higher human-evaluation win rate than pre-trained Wan2.1 for text-to-video and image-to-video.The comparison used 20 pairs of 720p videos per task, each assigned to 30 users; higher brightness and greater motion were preferred by average users.Source: §D.2 · pp. 21–22, Table 10 · p. 22

D.3. Image Editing

  • For 1K image editing, DC-Gen-Qwen-Image-Edit approaches the pre-trained model in visual effects and text alignment for English and Chinese prompts.The human study used 50 GEdit samples and 20 users per sample.Source: §D.3 · p. 22, Table 11 · p. 23

E. Hyperparameters

  • The three-stage pipeline trains at 512px for image tasks and 480p for video, then applies end-to-end LoRA fine-tuning at higher resolutions.The higher-resolution stage is termed resolution lifting; standard configurations are 1K for image tasks and 720p, 80 frames for video.Source: E. Hyperparameters · p. 24, Table 12 · p. 24, Table 13 · p. 25

F.1. Training Cost

DC-Gen post-training takes fewer than 40 H100 GPU days across models, with embedding alignment accounting for a small share of training cost. Its relative inference speedup grows with resolution, and it can be combined with orthogonal acceleration methods.

F.1. Training Cost

  • DC-Gen post-training for all models is completed within 40 H100 GPU days, and embedding alignment accounts for only a small proportion of total training cost.The authors state that alignment improves stability, quality, and convergence speed of end-to-end fine-tuning at minimal additional cost.Source: §F.1 · p. 26, Table 15 · p. 26, Figure 21 · p. 26

F.2. Inference Speedup

  • DC-Gen’s relative inference speedup increases with resolution because visual tokens grow while text-token counts remain constant.Latency is simplified as transformer-backbone cost; for targeted high-resolution generation, autoencoder decoding is nearly negligible by comparison.Source: §F.2 · pp. 26–27, Figure 22 · p. 26, Table 16 · p. 27

G. Combine DC-Gen with Orthogonal Techniques

  • DC-Gen’s token-count reduction is orthogonal to inference accelerators and can be combined with quantization, feature caching, and few-step distillation.The paper reports combination experiments with SVDQuant, TaylorSeer, and AnyFlow on FLUX.1-Krea at 1024px.Source: G. Combine DC-Gen with Orthogonal… · p. 27, Table 17 · p. 27

H. Deep Compression Video Autoencoder (DC-AE-V)

DC-AE-V uses chunk-causal temporal modeling to balance reconstruction quality under deep compression with generalization to longer videos. It outperforms prior autoencoders on reported reconstruction comparisons and gives Wan2.1 faster video generation with slightly higher VBench scores.

Video Autoencoder Design Trade-offs

  • Under deep compression, causal video autoencoders have limited reconstruction accuracy, while non-causal models reconstruct better but generalize poorly to longer videos.Temporal tiling and blending can partly alleviate the non-causal generalization issue but may introduce flickering and boundary blurring.Source: H. Deep Compression Video Autoencoder… · pp. 27–28, Figure 25 · p. 29

Chunk-Causal Temporal Modeling

  • Chunk-causal modeling uses bidirectional information within fixed-size chunks and causal flow across chunks, balancing reconstruction quality with longer-video generalization.Unlike grouped causal convolution, chunk size is independent of temporal compression; quality improves as chunk size increases, with a final size of 40 where benefits plateau as training costs rise.Source: H. Deep Compression Video Autoencoder… · pp. 27–28, Figure 23 · p. 28

Video Reconstruction Results

  • At the same compression ratio, DC-AE-V reconstructs better than causal LTX Video VAE and non-causal Video DC-AE, while generalizing better to longer videos than Video DC-AE.Video DC-AE can reach higher PSNR with tiling and blending but still generalizes poorly to longer videos.Source: H. Deep Compression Video Autoencoder… · p. 28, Table 18 · p. 29

Video Generation Results

Model Architecture

  • The f32t4c32 DC-AE-V architecture has six encoder and decoder stages, with spatial resampling in the first five and temporal resampling in the last.It uses 3D ResNet blocks, Residual Autoencoding, and a video-capable extension of the StyleGAN2 discriminator.Source: H. Deep Compression Video Autoencoder… · p. 29, Figure 26 · p. 31

Efficiency

Loading 2509.25180v3…