PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Diffusion Model without Variational Autoencoder

Minglei Shi, Haolin Wang, Wenzhao Zheng, Ziyang Yuan, Xiaoshi Wu, Xintao Wang, Pengfei Wan, Jie Zhou, Jiwen Lu

arXiv:2510.15301v4cs.CVcs.AI

TL;DR

The paper addresses inefficiency and limited task transfer in VAE-based latent diffusion, linking these problems to insufficient semantic structure in VAE latents. It proposes SVG, which combines frozen DINO features with a residual detail branch, and reports faster training, few-step sampling, and improved generative quality. The authors position SVG as a task-general representation, while noting unresolved efficiency, guidance, and scaling limitations.

  • Problem

    VAE+Diffusion systems have limited training efficiency, slow inference, and poor transferability because their latent spaces lack clear semantic separation and discriminative structure.

  • Method

    SVG trains diffusion models directly in a feature space built from frozen DINOv3 features augmented with lightweight residual features for fine-grained details.

  • Results

    SVG-XL achieves rFID=0.65 and, with 25-step generation after 80 training epochs, reaches gFID=6.57 (w/o CFG) and gFID=3.54 (w/ CFG), outperforming the reported baselines.

  • Takeaways & Limitations

    SVG provides a unified feature space that supports generation alongside diverse core vision tasks while preserving the semantic and discriminative capabilities of self-supervised representations.

  • Takeaways & Limitations

    SVG features remain high-dimensional, classifier-free guidance is less effective, and performance on larger datasets, higher resolutions, and challenging T2I/T2V tasks remains underexplored.

Abstract

from arXiv · show

Recent progress in diffusion-based visual generation has largely relied on latent diffusion models with variational autoencoders (VAEs). While effective for high-fidelity synthesis, this VAE+diffusion paradigm suffers from limited training efficiency, slow inference, and poor transferability to broader vision tasks. These issues stem from a key limitation of VAE latent spaces: the lack of clear semantic separation and strong discriminative structure. Our analysis confirms that these properties are crucial not only for perception and understanding tasks, but also for the stable and efficient training of latent diffusion models. Motivated by this insight, we introduce SVG, a novel latent diffusion model without variational autoencoders, which leverages self-supervised representations for visual generation. SVG constructs a feature space with clear semantic discriminability by leveraging frozen DINO features, while a lightweight residual branch captures fine-grained details for high-fidelity reconstruction. Diffusion models are trained directly on this semantically structured latent space to facilitate more efficient learning. As a result, SVG enables accelerated diffusion training, supports few-step sampling, and improves generative quality. Experimental results further show that SVG preserves the semantic and discriminative capabilities of the underlying self-supervised representations, providing a principled pathway toward task-general, high-quality visual representations. Code and interpretations are available at https://howlin-wang.github.io/svg/.

1 INTRODUCTION

The paper identifies semantically entangled VAE latents as a bottleneck for efficient latent diffusion and proposes SVG, which combines discriminative DINO features with residual detail modeling. SVG is presented as enabling rapid training, efficient inference, and strong generation quality while supporting broader vision tasks.

  • Standard VAE+Diffusion training and inference are computationally expensive, requiring 7M training steps and typically more than 25 sampling steps for ImageNet 256 × 256 generation.
  • SVG uses self-supervised DINOv3 features to construct a discriminative feature space while retaining strengths beyond generation.
  • Semantic entanglement in vanilla VAE latents is identified as a major obstacle to efficient diffusion and motivates semantically structured feature spaces.
  • A lightweight Residual Encoder augments frozen DINOv3 features with fine-grained perceptual details, producing a unified representation for reconstruction and transferability.
  • SVG Diffusion combines impressive generative quality with rapid training and highly efficient inference.

2 RELATED WORKS

The related-work discussion contrasts generative modeling paradigms with visual representation learning and highlights the difficulty of supporting diverse vision tasks in one representation. SVG repurposes self-supervised features for generation to construct such a unified feature space.

  • Visual generation: Generative modeling includes adversarial and autoregressive approaches, while diffusion models have emerged as a dominant paradigm for visual generation.
  • Visual representation learning: Self-supervised visual representation methods learn informative features that support zero-shot learning, retrieval, and semantic understanding.
  • Visual representation learning: Existing visual representations often struggle to provide a unified solution across diverse visual tasks.
  • Visual representation learning: SVG demonstrates that self-supervised features can be directly repurposed for generative modeling and support diverse core vision tasks.

3 METHODOLOGY

SVG rethinks latent diffusion by using semantically structured DINOv3 features, supplemented with residual details for reconstruction, instead of conventional VAE latents. Its analysis links semantic separation to simpler diffusion dynamics, while its architecture and two-stage training preserve semantics and improve perceptual fidelity.

  • 3.1 PRELIMINARIES: Diffusion models transform Gaussian noise toward the data distribution through iterative inference, using either reverse-time dynamics or a probability-flow ODE.The diffusion process interpolates noisy observations between data and Gaussian noise, and training uses a denoising objective.
  • 3.2 RETHINKING LATENT DIFFUSION MODELS: VAE latent spaces can be computationally costly and semantically entangled, with better reconstruction often conflicting with better generation.The paper motivates alternatives by noting expensive training and inference, difficult control of perceptual compression, and mixed representations across semantic classes.
  • 3.2 RETHINKING LATENT DIFFUSION MODELS: Clear class separation produces consistent within-class velocity directions and divergent between-class directions, simplifying optimization and enabling high-quality results with fewer sampling steps.The toy visualization contrasts structured dynamics with entangled latent spaces, where velocity directions are less consistent across semantic classes.
  • 3.3 VISUAL FEATURE GENERATION: SVG combines frozen DINOv3 features for semantic discriminability with a lightweight Residual Encoder that captures fine-grained perceptual information for faithful reconstruction.Residual outputs are combined with DINOv3 features, and omitting the residual branch reduces reconstruction quality, especially for color and high-frequency details.
  • 3.3 VISUAL FEATURE GENERATION: SVG Diffusion trains directly on the high-dimensional combined feature space using flow matching, rather than on a low-dimensional VAE latent space.For 256×256 images, the DINOv3 feature map is 16×16×384, compared with DiT’s 16×16×4 VAE latent.
  • 3.3 VISUAL FEATURE GENERATION: The two-stage pipeline trains reconstruction components first, aligns residual features to DINOv3 batch statistics, then trains diffusion with normalization for stable optimization.The residual alignment matches feature means and standard deviations, while the second stage applies QK-Norm and per-channel feature normalization.

4 EXPERIMENTS

Experiments evaluate SVG’s generation quality, efficiency, scaling, task generalization, encoder design, editing, and interpolation. SVG delivers strong few-step generation, preserves downstream capabilities, and supports coherent edits and smooth interpolations.

  • Main results: SVG-XL achieves gFID=6.57 without CFG and gFID=3.54 with CFG after 25-step generation and 80 training epochs.With 500 epochs, it reaches gFID=3.94 without CFG and gFID=2.10 with CFG.
  • Analysis: SVG achieves substantially better performance than SiT under few-step sampling and maintains an advantage across model capacities.SVG reaches substantially lower FIDs with 10 steps, while SiT requires 250 steps for reasonable FIDs.
  • Task generalization: SVG features preserve DINO’s generalization on ImageNet-1K classification, ADE20K segmentation, and NYUv2 depth estimation.Results are comparable or slightly superior using lightweight decoders on frozen encoders.
  • Encoder analysis: The Residual Encoder improves DINO-based reconstruction by capturing fine-grained details, but naive feature concatenation imbalances the distribution and harms generative modeling.Comparisons indicate SVG balances semantic representation with satisfactory reconstruction, unlike SigLIP2 or MAE alone.
  • Generalization: SVG generates coherent zero-shot class-conditioned edits while preserving non-edited regions.The evaluation replaces selected image regions with noise before class-conditioned sampling.
  • Generalization: SVG produces smooth, high-quality images under both direct linear and spherical linear interpolation.The results support continuity and robustness of the feature space under moderate deviations from the training distribution.

5 CONCLUSION

The paper identifies semantically undiscriminable latent structure as a limitation for efficient latent diffusion and proposes SVG, which enriches frozen DINO features with residual features. The resulting unified space supports efficient generation and diverse vision tasks, while larger-scale and harder applications remain underexplored.

  • Semantically undiscriminable latent structure is identified as a key factor limiting latent diffusion training and inference efficiency.
  • SVG enriches frozen DINO features with residual features that capture fine-grained perceptual details.
  • The unified SVG feature space supports faster diffusion training, few-step sampling, improved generation, and diverse core vision tasks.
  • Further work includes reducing feature dimensionality, refining the residual encoder, improving guidance alternatives, and evaluating larger or more challenging tasks.

A MORE IMPLEMENTATION DETAILS

The appendix reports the hyperparameter setup, compute resources, sampler, and classifier-free guidance procedure used for the experiments.

  • Architecture, optimization, and interpolant hyperparameters are reported in Table 5.
  • Reconstruction and generation experiments are trained on 8×H100 GPUs.
  • Euler’s method is used as a first-order ODE sampler, so sampling steps directly correspond to function evaluations.
  • For classifier-free guidance, the paper reports FID50K with cfg 1.5 and uses zero-init to skip the first step.

B FINETUNING DETAILS ON DOWNSTREAM TASKS

Downstream evaluation compares frozen SVG and DINOv3 encoders on classification, semantic segmentation, and depth estimation using lightweight task-specific decoders.

  • The study evaluates ImageNet-1K classification, ADE20K semantic segmentation, and NYUv2 depth estimation with frozen encoders.
  • Image classification: Image classification uses a linear classifier and reports Top-1 and Top-5 accuracy.
  • Semantic segmentation: Semantic segmentation uses an FPNHead decoder and reports mean Intersection-over-Union and mean Accuracy.
  • Depth estimation: Depth estimation regresses pixel-wise depth and reports Absolute Relative Error and RMSE.

C EDITING DETAILS

The editing procedure uses SDEdit-style trajectory inversion to preserve spatial and semantic consistency while modifying an input image.

  • The method follows an SDEdit-style procedure with trajectory inversion for editing experiments.
  • The input image is inverted into the diffusion latent space before editing begins.
  • The noisy inversion trajectory is recorded through a target editing timestep to provide a reference for preserved regions.

D LATENT SPACE INTERPOLATION TEST

SVG’s latent space is evaluated through linear and spherical interpolation of Gaussian noise vectors under a shared class embedding. The resulting images remain smooth and natural, including under out-of-distribution linear interpolation.

  • The test randomly samples two noise vectors from a standard Gaussian distribution and conditions interpolants on the same class embedding.
  • Linear interpolation and spherical linear interpolation are both used to assess continuity in the SVG feature space.
  • Slerp better preserves vector norms and is less likely to produce interpolants outside training distributions.
  • SVG outputs vary smoothly under slerp, while direct linear interpolation still produces natural, high-quality images despite being out-of-distribution relative to training.

E ADDITIONAL COMPARISONS OF DIFFERENT LATENT SPACE

Additional comparisons show that distribution alignment is important for SVG’s semantic discriminability and generation quality, while encoder-layer and tokenizer analyses characterize its feature space and efficiency.

  • Removing distribution alignment weakens SVG’s semantic discriminability and degrades generative performance.
  • The encoder-layer study evaluates reconstruction quality with rFID and generative quality with gFID under identical diffusion-model settings.
  • The comparison includes t-SNE visualizations using 100 ImageNet classes or 20 ImageNet classes, with each class assigned a distinct color.
  • SVG Autoencoder has parameter scale and end-to-end latency comparable to prior baselines, while its encoder FLOPs are substantially lower than existing VAE-based methods.

G FURTHER ANALYSIS OF SVG GENERATION

Further analysis finds that SVG produces cleaner, more structured feature representations than a vanilla VAE-based DiT model and scales to higher resolutions without architectural changes.

  • SVG produces cleaner and more structured feature maps than the VAE-based DiT model, especially at large timesteps.
  • SVG hidden states exhibit more discriminative characteristics that benefit generation quality and downstream tasks.
  • SVG reconstruction and generation experiments extend to 512×512 and 1024×1024 resolutions.
  • The method scales to higher resolutions without requiring architectural modifications.

I MORE QUALITATIVE RESULTS

Additional results report SVG’s feature-map cleanliness, stable convergence, classifier-free guidance setting, and qualitative ImageNet generation across resolutions and classes.

  • SVG produces cleaner feature maps than the VAE-Diffusion model, which becomes particularly noisy at large t.
  • SVG-XL qualitative evaluations include random ImageNet 256×256 samples and uncurated generations for specific classes.The reported results are described as demonstrating diversity and visual quality.
  • All SVG model scales, including B, L, and XL, demonstrate similarly stable convergence behavior during training.
  • SVG-XL generation results are shown at 512×512 and 1024×1024 resolutions using classifier-free guidance with w = 4.0.
Loading 2510.15301v4…