PaperlayerView exact revision on arXiv

Source-linked AI summary

DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence

Xu Huang, Ye Huang, Zijun Liao, Yuwei Niu, Xiaojie Li, Menghan Zhou, De Wen Soh, Xiaotong Li, Daquan Zhou

arXiv:2609.39222v1cs.CV

TL;DR

High compression can shorten latent sequences and reduce per-pass cost, but may slow diffusion convergence, while semantic encoders can lose pixel-level detail needed for faithful reconstruction. DC-SAE combines a compact semantic branch with an unconstrained pixel branch, and at 32× compression reports stronger ImageNet reconstruction and generation metrics than DC-AE.

  • Problem

    High-compression tokenizers reduce latent sequence length but can require more diffusion training steps, while semantic encoders may omit low-level details needed for faithful reconstruction.

    Source: §1 · pp. 1–2

  • Method

    DC-SAE pairs a compact semantic encoder with an unconstrained pixel branch, combining their latents for decoding with decoder-side Spatial DeMerger expansion.

    Source: §5 · p. 11

  • Results

    At 32× on ImageNet 512 × 512, DC-SAE achieves 29.79 PSNR and 3.37 gFID versus DC-AE’s 26.25 PSNR and 7.47 gFID.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ImageNet experiments show DC-SAE combines strong reconstruction, faster diffusion convergence, and competitive generation performance at high compression.

    Source: §5 · p. 11

    A semantic branch alone is insufficient for high-compression visual tokenization because increasing compression amplifies its loss of pixel-level detail.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

High-compression tokenizers are essential for scaling latent image generative models. However, aggressive compression creates a fundamental tradeoff between reconstruction fidelity and generation efficiency: high compression image encoder always increases the learning difficulty of diffusion training, resulting in slow model convergence. Recent representation autoencoders speed up the diffusion training by improving the latent feature's expressive capability by replacing VAE encoders with pretrained semantic encoders, yet they are typically limited to moderate compression and lose pixel-level details necessary for faithful reconstruction. To achieve both high compression and fast diffusion training, we propose DC-SAE, a Decoupled Compact Semantic Autoencoder designed for high-compression image generation with accelerated diffusion model convergence. DC-SAE consists of two key components: (1) a macro-level architecture design that leverages semantic encoders to enable higher compression ratios, and (2) a pixel-level encoder that preserves low-level details, ensuring high-fidelity image reconstruction. We empirically demonstrate that DC-SAE performs strongly on image generation tasks, achieving both compact latent representations and efficient training dynamics. Specifically, on the ImageNet dataset with $512 \times 512$ resolution, DC-SAE achieves $32\times$ spatial compression, with 29.79 PSNR and 3.37 gFID, substantially outperforming the previous state-of-the-art high-compression tokenizer baselines DC-AE by 13.5% and 54.9% on PSNR and gFID, respectively, maintaining comparable throughput and faster diffusion model training convergence. Beyond class-conditional generation, a $1.6$B-parameter DiT using DC-SAE achieves 0.84 on GenEval and 86.007 on DPG-Bench for text-to-image generation at $1024\times1024$ resolution.

1 Introduction

The introduction frames high compression as reducing latent cost while potentially slowing diffusion convergence, and presents DC-SAE’s semantic–pixel design as a way to combine compact structure with reconstruction detail. It reports improved quality and throughput at 32× compression.

1 Introduction

  • High-compression tokenizers shorten latent sequences and reduce per-pass cost, but diffusion models may need more training steps to converge, reducing overall training-cost benefits.The introduction asks how to retain high compression while achieving fast diffusion convergence.Source: §1 · pp. 1–2
  • At 32× on ImageNet 512 × 512, DC-SAE reaches 29.79 PSNR and 3.37 gFID versus DC-AE’s 26.25 PSNR and 7.47 gFID.Figure 1 compares reconstruction details and gFID; the reported metrics quantify the comparison.Source: Figure 1 · p. 2, §1 · p. 2
  • The semantic branch supplies compact, structured features while the unconstrained pixel branch restores fine details; Spatial DeMerger expands the fused latent only on the decoder side.The two branches together outperform semantic-only and pixel-only latents in diffusion convergence under 32× compression.Source: §1 · pp. 2–3
  • At 256 × 256, 512 × 512, and 1024 × 1024, DC-SAE has 4.64×, 4.75×, and 4.41× higher throughput than DC-AE, respectively.Figure 1 also highlights higher autoencoder throughput.Source: §1 · p. 2, Figure 1 · p. 2
  • With DC-SAE, a 1.6B-parameter DiT using a Qwen3-1.7B text encoder at 1024 × 1024 with classifier-free guidance achieves 0.84 on GenEval and 86.007 on DPG-Bench.Source: §1 · p. 2
  • DC-SAE adapts pretrained semantic autoencoders from 16× to 32× compression without retraining, including encoders from DINO and CLIP series [4] [30] [38] [34] [50].Source: §1 · p. 3

2 Related work

Related work spans pretrained semantic encoders and unified tokenizers, alongside deep-compression methods that balance compact latents against reconstruction detail. DC-SAE combines semantic and pixel information to extend semantic autoencoders toward higher compression.

2.1 Representation encoders and unified tokenizers for generation

  • Pretrained representation encoders provide semantically structured features and can accelerate diffusion-transformer training, but their objectives may discard texture, local geometry, color statistics, and high-frequency detail [13] [30] [34] [50] [52].Source: §2.1 · p. 3
  • Unified tokenizer work adapts visual representations through decoupled or multi-codebook designs, self-distillation, pixel-level decoding, and shared semantic spaces [26] [33] [39] [47] [17].Other general-purpose tokenizers span image, video, and multimodal tasks [9] [16] [25].Source: §2.1 · p. 3
  • SVG adds a lightweight residual branch to frozen DINO features [37], whereas DC-SAE targets high compression with an unconstrained pixel encoder and reports better reconstruction than SVG and UniLiP [39].DC-SAE does not rely on SVG-style self-distillation or semantic regularization and retains the fast convergence of semantic latents.Source: §2.1 · p. 3

2.2 Deep-compression visual tokenizers

  • Deep-compression visual tokenizers pursue shorter generator sequences through approaches including residual autoencoding [6] [7], text-guided tokenization [49], masked modeling [5], and staged token compression [21].Other directions include video autoencoding, diffusion-guided decoding, adaptive token allocation, and tokenizer evaluation [23] [29] [43] [53] [2] [42] [51].Source: §2.2 · p. 4
  • DC-SAE separates compact semantic latents for generation from pixel features for reconstruction, then uses decoder-side Spatial DeMerger expansion to restore denser spatial layout without lengthening the generator sequence.Figure 2 depicts the semantic and pixel branches fused for decoding.Source: §2.2 · p. 4, Figure 2 · p. 4

3 Method

DC-SAE targets 32× semantic compression while retaining diffusion-friendly structure and reconstruction fidelity. It combines semantic and pixel streams and separates the compact diffusion grid from the denser decoder grid.

3 Method

  • The method investigates whether semantic autoencoders commonly used at 16× compression can be scaled to 32× to improve throughput.The paper's construction resizes the input to half resolution before feeding it to the frozen semantic encoder.Source: §3 · p. 5, §3.1 · p. 5

3.1 Limitations of Semantic Autoencoders under Deep Compression Regime

  • Half-resolution input resizing creates a 32× semantic latent, but at 256 × 256 it yields only around 15 PSNR and 7.15 gFID after 80 DiT epochs.The result indicates that this direct strategy does not retain the expected fast diffusion convergence at deep compression.Source: §3.1 · p. 5
  • Post-merge alternatives, including average pooling and token mergers, perform worse than resizing-based pre-merge compression.This suggests the bottleneck is not merely the choice of compression implementation.Source: §3.1 · p. 5
  • Semantic-only compression loses pixel detail because recognition-oriented features are not trained for invertible reconstruction, and this loss is amplified from 16× to 32×.Discarded information includes color statistics, textures, local boundaries, and high-frequency patterns.Source: §3.1 · p. 5

3.2 Semantic-guided Unconstrained AutoEncoder

  • DC-SAE pairs a frozen semantic encoder with an unconstrained pixel encoder, concatenates their latents, and applies sample-wise pixel-latent dropout during joint decoding.The semantic branch supplies object-level layout and relations, while the pixel branch focuses on texture, color, boundaries, and high-frequency information.Source: §3.2 · p. 6
  • At 80 epochs, joint semantic–pixel reaches 4.27 gFID, versus 7.15 for semantic-only and 11.04 for pixel-only latents.The joint latent combines the semantic branch's structure with the pixel branch's reconstruction-critical details.Source: §3.2 · p. 6

3.3 ViT Decoder with Spatial Demerger

  • Spatial DeMerger expands compact tokens only before the ViT decoder, keeping DiT on the sparse 32× grid while giving reconstruction a denser grid.This decouples the grid used for diffusion from the grid used for decoding.Source: §3.3 · p. 6

4 Experiments

DC-SAE is evaluated on class-conditional ImageNet and text-to-image generation, with ablations testing its semantic–pixel design and high-compression choices. Results cover reconstruction, generation quality, and component roles.

4.2 Main Result

  • At 32× compression on ImageNet 512 × 512, DC-SAE achieves 3.37 gFID; at 256 × 256, it reaches 0.45 rFID, 3.31 gFID, and 180.71 Inception Score.The 256 × 256 rFID improves from 0.69 in the comparison described.Source: §4.2 · pp. 7–8, Table 2 · p. 8, Table 5 · p. 9

4.3 Text-to-Image Generation

  • The text-to-image model uses a 1.6B DiT with Qwen3-1.7B, trains progressively at 256 × 256 and 512 × 512, then undergoes SFT at 1024 × 1024.Training uses a subset of internal data together with LAION-COCO; evaluation applies CFG on GenEval and DPG-Bench.Source: §4.3 · p. 8
  • At 1024 × 1024, the 1.6B-parameter DiT scores 0.84 on GenEval and 86.007 on DPG-Bench, leading the cited GenEval baselines and remaining competitive on DPG-Bench.The cited DC-AE-based 12B model scores higher on DPG-Bench, at 87.073.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 Ablation Studies

  • At 16× compression, DC-SAE reaches 27.80 PSNR and 0.26 rFID versus RAE’s around 19 PSNR, while retaining strong ImageNet generation after 80 epochs.The test isolates the pixel branch because pretrained semantic encoders already provide a sufficiently dense token grid at 16×.Source: §4.4 · p. 9
  • Post-merge variants underperform: reducing token count after semantic extraction loses information, while input-level pre-merge compression yields better generation quality at 32×.The comparison includes average pooling, frozen mergers, and learnable mergers against the resizing-based pre-merge strategy.Source: Table 4 · p. 9, §4.4 · p. 10
  • At 32×, Spatial DeMerger improves validation PSNR over No-DeMerger by giving the decoder a denser grid while diffusion operates on the sparse latent.The comparison uses the standard DINOv2 pre-merge setting on ImageNet 256 × 256.Source: §4.4 · p. 10
  • Late concatenation underperforms joint semantic–pixel autoencoding, indicating that the gain is not simply from giving DiT semantic features.DiT receives semantic features in both settings; only Joint learns the coupled latent representation during autoencoding.Source: §4.4 · p. 10, Table 6 · p. 10

4.5 Discussion: Understanding Component Contributions to Reconstruction Fidelity

  • Masking the pixel branch severely degrades reconstruction, whereas masking the semantic branch has little impact; the branches supply complementary reconstruction and generative structure.The pixel branch preserves texture, color, and spatial detail; the semantic branch provides high-level structure and a generation-friendly representation.Source: §4.5 · p. 11

5 Conclusion

DC-SAE combines a frozen semantic encoder with an unconstrained pixel branch to balance compact, generation-friendly representations with reconstruction detail. The conclusion reports strong high-compression ImageNet and text-to-image results.

5 Conclusion

  • Masking the pixel component severely degrades reconstruction, while masking the semantic component has little effect.This comparison indicates that reconstruction relies primarily on the pixel branch.Source: Figure 5 · p. 11
  • At 32× compression on ImageNet 512 × 512, DC-SAE achieves 29.79 PSNR and 3.37 gFID; its 1.6B-parameter DiT scores 0.84 on GenEval and 86.007 on DPG-Bench at 1024 × 1024.The conclusion presents these results as evidence of strong reconstruction, generation, and text-to-image applicability.Source: §5 · p. 11
  • DC-SAE addresses the reconstruction–generation tradeoff with a frozen semantic encoder for generation-friendly structure and an unconstrained pixel branch for low-level reconstruction details.Input-level semantic compression and a decoder-side Spatial DeMerger support high-compression tokenization with strong reconstruction fidelity.Source: §5 · p. 11

A.1 Limitations and Future Work.

The evaluation and analysis leave boundaries around the breadth of text-to-image validation and the understanding of semantic encoder properties.

A.1 Limitations and Future Work.

  • Text-to-image evaluation uses one 1.6B DiT configuration and a subset of internal data with LAION-COCO; broader scale, dataset, and architecture validation remains future work.The paper also says which semantic-encoder properties matter for generation, and when pre- versus post-merge compression is preferable, remain insufficiently understood.Source: §A.1 · p. 15

A.2 Tokenizer Training Details

The tokenizer combines a frozen semantic encoder with trainable pixel and decoder components, then trains at 256 × 256 before 512 × 512 fine-tuning.

A.2 Tokenizer Training Details

  • ImageNet-1K is used for tokenizer training; most experiments use 256 × 256 images, and 512 × 512 training starts from the 256 × 256 checkpoint.Source: §A.2 · p. 15
  • The tokenizer uses a frozen semantic encoder, trainable pixel encoder and fusion projection, and a ViT decoder with Spatial DeMerger.The pixel encoder, fusion projection, Spatial DeMerger, and decoder are trained for reconstruction; decoder and discriminator largely follow RAE.Source: §A.2 · p. 15
  • Tokenizer training uses L1 and LPIPS loss weights of 1 and adversarial loss weight of 0.5, with 30 epochs at 256 × 256 and 20 more at 512 × 512.Both stages use batch size 512; the 512 × 512 stage initializes from the converged 256 × 256 checkpoint.Source: §A.2 · p. 15

A.3 Throughput Comparison with DC-AE

Under the reported measurement settings, DC-SAE achieves higher end-to-end throughput than DC-AE at every evaluated resolution, with speedups above 4× throughout.

A.3 Throughput Comparison with DC-AE

  • DC-SAE’s end-to-end throughput exceeds the corresponding DC-AE baseline by more than 4× at every evaluated resolution.Table 7 compares throughput in images per second.Source: §A.3 · p. 15, Table 7 · p. 15
  • Throughput was measured at batch size 32 on one NVIDIA H200 GPU using bf16.The protocol used 10 warmup iterations and 50 measurement iterations.Source: §A.3 · p. 15
  • The comparison reports end-to-end throughput in images per second for DC-AE and DC-SAE.The measurements use the same settings for both methods.Source: Table 7 · p. 15, §A.3 · p. 15

A.4 Additional Analysis of Spatial DeMerger

Additional QwenViT-based experiments show that Spatial DeMerger improves reconstruction quality with two frozen semantic-encoder configurations and no pixel encoder, a setting relevant to RAE-like systems.

A.4 Additional Analysis of Spatial DeMerger

  • The two tested configurations are native QwenViT merger and simple 2 × 2 average pooling for reducing vision-token spatial resolution.The native merger follows the Qwen-VL design, with a 16× patch-based QwenViT and a 2 × 2 merger trained with the MLLM.Source: §A.4 · p. 15
  • Spatial DeMerger consistently improves reconstruction quality with both the original QwenViT merger and a 2 × 2 average-pooling variant.Both configurations use frozen semantic encoders and no additional pixel encoder.Source: §A.4 · p. 16, Table 8 · p. 16
  • The experiments keep the semantic encoder fully frozen and omit the pixel encoder, requiring the decoder to recover spatial details from semantic representations alone.This isolates Spatial DeMerger under a stricter reconstruction setting than the main-paper experiments.Source: §A.4 · p. 16, Table 8 · p. 16
  • The results suggest Spatial DeMerger is especially useful in RAE-like settings with sparse semantic tokens and no auxiliary pixel-level features.Source: §A.4 · p. 16
Loading 2609.39222v1…