PaperlayerView exact revision on arXiv

Source-linked AI summary

VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models

Tianci Bi, Xiaoyi Zhang, Yan Lu, Nanning Zheng

arXiv:2510.18457v3cs.CVcs.LG

TL;DR

LDM performance depends heavily on visual-tokenizer quality, yet distilling VFM representations can weaken their robustness. VFM-VAE bypasses distillation with a frozen VFM encoder and specialized decoder, reaching 1.62 gFID (w/o CFG) after 640 epochs while improving diffusion training efficiency.

  • Problem

    Distillation-based VFM alignment can degrade representation robustness under semantic-preserving transformations, limiting visual-tokenizer quality for LDMs.

    Source: §1 · p. 2

  • Method

    VFM-VAE directly uses a frozen VFM encoder with a specialized multi-scale decoder and objectives balancing semantic preservation with reconstruction fidelity.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5

  • Results

    1.62 gFID (w/o CFG) is achieved in 640 epochs, while joint tokenizer–diffusion alignment reaches 2.22 gFID (w/o CFG) in 80 epochs.

    Source: §1 · p. 3, §4.4 · p. 8

  • Takeaways & Limitations

    Frozen VFMs can serve as practical visual tokenizers for LDMs while preserving native semantics and supporting faster diffusion training.

    Source: §5 · p. 8

    VFM-VAE sacrifices some high-frequency fidelity, complicates tuning through inherited objectives, and makes SE-CKNNA scores relative to the aligned VFM.

    Source: §5 · p. 8

Abstract

from arXiv · show

The performance of Latent Diffusion Models (LDMs) is critically dependent on the quality of their visual tokenizers. While recent works have explored incorporating Vision Foundation Models (VFMs) into the tokenizers training via distillation, we empirically find this approach inevitably weakens the robustness of learnt representation from original VFM. In this paper, we bypass the distillation by proposing a more direct approach by leveraging the frozen VFM for the LDMs tokenizer, named VFM Variational Autoencoder (VFM-VAE).To fully exploit the potential to leverage frozen VFM for the LDMs tokenizer, we design a new decoder to reconstruct realistic images from the semantic-rich representation of VFM. With the proposed VFM-VAE, we conduct a systematic study on how the representation from different tokenizers impact the representation learning process throughout diffusion training, enabling synergistic benefits of dual-side alignment on both tokenizers and diffusion models. Our effort in tokenizer design and training strategy lead to superior performance and efficiency: our system reaches a gFID (w/o CFG) of 2.22 in merely 80 epochs (a 10$\times$ speedup over prior tokenizers). With continued training to 640 epochs, it further attains a gFID (w/o CFG) of 1.62. These results offer solid evidence for the substantial potential of VFMs to serve as visual tokenizers to accelerate the LDM training progress.

1. Introduction

VFM-VAE directly uses frozen VFM encoders instead of distilling their representations, pairing them with a specialized decoder to preserve semantics while reconstructing images. The resulting tokenizer and alignment strategy improve diffusion representation learning, convergence, and generation quality.

Evaluation

  • SE-CKNNA evaluates tokenizer–VFM alignment under semantic-preserving perturbations, exposing robustness differences that standard CKNNA can miss.The metric averages alignment across semantic-preserving transformations.Source: Figure 2 · p. 2

Motivation

  • Distillation-based alignment can degrade VFM representations and make them brittle under semantic-preserving transformations.This motivates directly retaining the original VFM features rather than training a VAE to mimic them.Source: §1 · p. 2

Proposed approach

  • VFM-VAE directly integrates a frozen VFM encoder into an LDM tokenizer and uses a specialized decoder for high-fidelity reconstruction.Its decoder uses multi-scale latent fusion and progressive-resolution reconstruction blocks to recover pixel-accurate images from semantically rich, spatially coarse features.Source: §1 · p. 2

Impact on diffusion models

  • High-quality tokenizer representations improve diffusion feature learning, while dual-side alignment strengthens VFM-aligned representations throughout diffusion models.The paper reports these effects across diffusion layers, with representation-aware supervision producing efficiency gains.Source: §1 · p. 2

Results

  • VFM-VAE reaches a gFID (w/o CFG) of 1.62 in 640 epochs, demonstrating strong performance and efficiency.The introduction presents this ImageNet 256 × 256 result as evidence for the approach’s effectiveness.Source: §1 · p. 3

2. Related work

Prior work improves LDM tokenizers or diffusion representations by aligning them with VFMs, but these approaches either distill and degrade VFM features or assume stable tokenizer latents. VFM-VAE instead preserves compressed VFM semantics while connecting tokenizer design to diffusion representation learning.

2.1. Visual tokenizers for latent diffusion

  • Visual tokenizers for LDMs incorporate VFM features through distillation, but alignment can degrade representations under semantic-preserving perturbations.VA-VAE and REPA-E align VAE or diffusion representations with VFM features, while other compression methods may sacrifice reconstruction fidelity.Source: §2.1 · p. 3

2.2. Representation alignment in diffusion models

  • Diffusion-alignment methods such as REPA and REG improve diffusion representations but assume the tokenizer already provides stable, semantically reliable latents.This paper instead studies how tokenizer representations influence diffusion-feature evolution.Source: §2.2 · p. 3

2.3. The relation with concurrent works

  • Unlike concurrent RAE and SVG methods that use high-dimensional VFM features directly, VFM-VAE preserves LDM latent compression for compatibility with existing diffusion frameworks.The paper also systematically examines how VFM-based encoders affect representation quality.Source: §2.3 · p. 3

3. Method

VFM-VAE directly uses a frozen VFM encoder and specialized decoder to preserve semantic representations while reconstructing high-fidelity images. The method further combines balanced training objectives with SE-CKNNA to evaluate robustness and study tokenizer effects on diffusion representations.

3. Method

  • VFM-VAE directly leverages a frozen VFM encoder, specialized learnable modules, and SE-CKNNA to preserve semantics while enabling high-fidelity reconstruction.The framework includes a multi-scale hierarchy, reconstruction decoder, balanced objective, and perturbation-aware alignment metric.Source: §3 · p. 3, §3.1 · p. 4

3.1. VFM-VAE encoder architecture

  • Multi-scale VFM features are concatenated and projected into a diagonal Gaussian posterior, reducing latent dimensionality while retaining information from shallow, middle, and final layers.The latent code is sampled with the reparameterization trick and constrained to preserve essential VFM information.Source: §3.1 · p. 4

3.2. VFM-VAE decoder architecture

  • The decoder combines global and spatial latent components with progressive reconstruction blocks that synthesize 256 × 256 images from 8 →16 →32 →64 →128 →256 resolutions.Global control is supplied across blocks, while spatial components are injected only into early low-resolution stages.Source: §3.2 · p. 4
  • Each decoder block receives direct multi-resolution supervision through ToRGB heads and feature-space residual connections, providing learning signals throughout hierarchical synthesis.The ToRGB heads combine newly refined details with stable structure from coarser stages.Source: §3.2 · p. 5

3.3. Training objectives

  • The training objective combines KL and VFM-based representation regularization with multi-resolution L1, adversarial, and LPIPS losses to balance semantic preservation, reconstruction fidelity, and realism.The adversarial loss uses a DINOv2-based discriminator, while LPIPS is intended to better match human perceptual similarity.Source: §3.3 · p. 5

3.4. Semantic-Equivariant CKNNA

  • SE-CKNNA averages CKNNA across semantic-preserving perturbations, addressing the brittleness of distillation-based tokenizers that can appear aligned on clean images.The method samples transformations such as additive noise, scale interpolation, and discrete rotations.Source: §3.4 · p. 5
  • Without explicit alignment, VFM-VAE yields higher average and peak layer-wise CKNNA than other tokenizer baselines, and alignment further improves its advantage.Figure 4 compares CKNNA across diffusion-model layers with and without explicit VFM alignment.Source: Figure 4 · p. 6

4. Experiments

Experiments show that VFM-VAE preserves semantic alignment while reconstructing images effectively, and that its latent representations improve diffusion learning and generation. Combining tokenizer-side and diffusion-side alignment yields faster convergence, stronger quality, and robust performance across settings.

4.2. Reconstruction and Alignment Evaluation

  • VFM-VAE preserves object structure and natural textures while surpassing SD-VAE on reconstruction FID and IS through its semantic–spatial dual-branch design.Source: §4.2 · p. 6
  • With 25% of VA-VAE’s training images, VFM-VAE’s SE-CKNNA differs from CKNNA by +1.6%, versus VA-VAE’s −33.2% drop, and linear probing improves from 31.9% to 43.2%.These results indicate more stable alignment and stronger linear-probe performance for VFM-VAE.Source: §4.2 · p. 6

4.3. Impact Analysis on Diffusion Models

  • Without explicit alignment, VFM-VAE-based diffusion models achieve higher average and peak CKNNA than VA-VAE-based models, with a peak of 0.52 versus the 0.50 VFM reference.LightningDiT trained with VFM-VAE attains 3.80 FID at 64 epochs without classifier-free guidance.Source: §4.3 · p. 7

4.4. Generation performance

  • ImageNet-512 generation and text-to-image evaluations report improved performance for VFM-VAE over VA-VAE, including with LightningDiT-B/1 and BLIP3-o.Source: Table 3 · p. 8, Table 4 · p. 8

4.3. Impact Analysis on Diffusion Models

  • Combining VFM-VAE with REG produces higher and more uniform layer-wise CKNNA across depths than REG or REPA alone.REG aligns shallow layers with VFM patch features and injects global semantics through the VFM class token.Source: §4.3 · pp. 7–8

4.4. Generation performance

  • 2.22 gFID (w/o CFG) in 64 epochs matches 480-epoch REG, while 1.62/1.31 FIDs (w/o and w/ CFG) at 640 epochs demonstrate faster convergence and superior quality.VFM-VAE + REG highlights the reported synergy between tokenizer-side and diffusion-side alignment.Source: Table 2 · p. 7, §4.4 · p. 8

4.5. Ablation studies

  • A minimal 5M-image baseline has 19.79 rFID; spatial control reduces it by 27%, later refinements reach 1.08, and final refinements push it below 1.0.Scaling an SD-VAE-style baseline instead decreases performance, while VFM-VAE remains compatible with different VFMs.Source: §4.5 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5. Conclusion and Limitations

The paper concludes that frozen VFMs can serve as practical visual tokenizers for latent diffusion while preserving native semantics. It also identifies reduced high-frequency fidelity, tuning complexity, and relative alignment scores as limitations.

5. Conclusion and Limitations

  • VFM-VAE directly integrates frozen VFMs into latent diffusion, avoiding representation degradation from distillation while retaining semantic alignment and reconstruction quality.Source: §5 · p. 8
  • Frozen VFMs sacrifice some high-frequency fidelity, and inherited complex training objectives complicate tuning.Source: §5 · p. 8
  • SE-CKNNA scores remain relative to the specific aligned VFMs used for comparison.Source: §5 · p. 8

Diffusion Models

The supplementary section positions VFM-VAE within latent diffusion, contrasts it with concurrent frozen-VFM tokenizers, and defines CKNNA and SE-CKNNA for evaluating representation alignment and robustness.

A. The relation with concurrent works

  • VFM-VAE preserves standard latent dimensions while using a specialized decoder to bridge semantic-rich frozen-VFM features and pixel-accurate synthesis.This supports compatibility with existing VAE-LDM infrastructure and avoids high-dimensional latent caching overhead.Source: A. The relation with concurrent works · p. 12

B.1. CKNNA formulation

  • CKNNA measures local neighborhood agreement between representations by restricting kernel comparison to mutual k-nearest-neighbor pairs.Higher CKNNA indicates stronger preservation of local neighborhood structure.Source: §B.1 · pp. 12–13

B.2. Semantic-Equivariant CKNNA formulation

  • SE-CKNNA averages CKNNA across semantic-preserving transformations, making alignment evaluation sensitive to robustness under perturbations.Its relative deviation from clean-image CKNNA reveals alignment stability, while inheriting the assumptions and limitations of CKNNA.Source: §B.2 · p. 13
  • The perturbation distribution samples additive noise, scale interpolations, and discrete rotations while preserving semantic content.Noise strengths are {0.05, 0.10, 0.15, 0.20}; scales are {0.25, 0.50, 0.75, 1.0}; rotations are {0°, 90°, 180°, 270°}.Source: §B.2 · p. 13

B.3. Evaluation setup

The architecture compresses hierarchical frozen-VFM features into standard latents and reconstructs images through global, spatial, and progressive-resolution decoder pathways.

B.3. Evaluation setup

  • Layer-wise alignment evaluation uses spatial tokens, global pooling, null conditioning, and diffusion timestep t = 0.5 in a unified protocol.The setup follows the cited REPA protocol and discards [CLS] tokens as in REG [46].Source: §B.3 · p. 13

C.1. Attention Projection

  • Attention Projection concatenates shallow, middle, and final VFM features, compresses them into sampled latent statistics, and expands them into decoder branches.PixelShuffle reconciles spatial mismatches before encoding and distributes decompressed outputs during decoding.Source: §C.1 · p. 13

C.2. Global and spatial branches

  • The decoder’s global branch supplies semantic style control, while its spatial branch hierarchically processes multi-resolution features for fine-grained detail.Together they combine holistic semantics with spatial information across progressive resolutions.Source: §C.2 · p. 14

C.3. Progressive reconstruction blocks

  • Progressive reconstruction blocks upsample features stage by stage, inject spatial latents during the first four stages, and use global modulation for final detail synthesis.Spatial latents reconstruct coarse and mid-level structure; later stages focus on high-frequency details.Source: §C.3 · p. 14

C.4. Upsampling modules

  • The upsampling module combines normalization, depthwise and pointwise convolutions, PixelShuffle, and Gaussian blur to increase resolution and suppress checkerboard artifacts.It also refines features before the ToRGB head.Source: §C.4 · pp. 14–15

D.1. Detailed reconstruction metrics

VFM-VAE delivers strong reconstruction across ImageNet resolutions, while its modular decoder is more stable and effective than a parameter-matched SD-VAE-like alternative.

D.1. Detailed reconstruction metrics

  • VFM-VAE achieves considerable LPIPS, PSNR, and SSIM reconstruction performance on ImageNet-256 and ImageNet-512 validation sets.The 512-resolution model is fine-tuned from 256-resolution pretrained weights.Source: §D.1 · p. 15
  • Removing the global pooled feature barely changes reconstruction but weakens generation, whereas scaling an SD-VAE-like baseline causes unstable or collapsed training.The results support the stability and effectiveness of VFM-VAE’s modular design.Source: §D.1 · p. 15

D.2. Detailed representation alignment metrics

Across matched and varied settings, VFM-VAE shows more robust alignment and stronger generation than VA-VAE, generalizes across VFMs and resolutions, and improves text-to-image results.

D.3. Not only one VFM choice

  • VFM-VAE remains effective across EVA-CLIP-Large, DINOv2-Large, and SigLIP2-Large, demonstrating architectural generality across VFM families.The authors adopt SigLIP2-Large as the default despite its non-optimal fit for ImageNet reconstruction.Source: §D.3 · pp. 15–16
  • DINOv2-Large leads generation, EVA-CLIP-Large leads most reconstruction metrics, and SigLIP2-Large offers a balance suited to alignment and text-to-image performance.VFM choice therefore creates a trade-off among reconstruction, generation, and text alignment.Source: §D.3 · pp. 15–16

D.4. Improvement is not due to a different VFM

  • Even under matched VFM and training conditions, VFM-VAE retains stronger perturbation-robust alignment and outperforms VA-VAE generatively, indicating the gain is architectural rather than merely due to a stronger VFM.The fair comparison uses SigLIP2-Large for both systems and still finds VA-VAE substantially behind.Source: §D.2 · p. 15, §D.4 · p. 16
  • VFM-VAE reaches stronger tokenizer alignment and reconstruction with fewer training images, while VA-VAE still lags when VFM, data, and diffusion configuration are matched.The same-VFM comparison shows that VA-VAE remains vulnerable to alignment drops under small input perturbations.Source: §D.4 · p. 16, Table 10 · p. 16

D.5. Preliminary generation on 512×512 resolution

  • VFM-VAE achieves clearly superior ImageNet-512 generation performance over VA-VAE after 100k training steps without CFG.The 512-resolution setup reuses most 256-resolution components and fine-tunes the Attention Projection module.Source: §D.5 · p. 17, Table 12 · p. 17

D.6. Text-to-image generation

  • With identical VLM and diffusion backbones, VFM-VAE raises DPG-Bench from 55.4 to 59.1 and lowers MJHQ-30K gFID from 23.00 to 16.98 after one epoch.The gains are strongest in global and relational understanding, with a slight entity-level decrease and a minor logo trade-off.Source: §D.6 · p. 17

E. Implementation details

The implementation uses staged VFM–VAE alignment and targeted fine-tuning, with adjusted diffusion training settings and reported efficiency comparisons. Additional evaluations cover text–image alignment and reconstruction quality at 256 resolution.

VFM-VAE training

  • VFM–VAE training uses strong alignment followed by weak alignment, reducing representation regularization to shift emphasis toward reconstruction quality.The recipe preserves stability under moderate weight changes when adversarial loss remains stable.Source: E. Implementation details · p. 17

Text-to-image evaluation

  • VFM-VAE + BLIP3-o achieves higher overall scores on DPG-Bench, indicating stronger text–image alignment at 256 resolution after one pretraining epoch.The table evaluates text-to-image generation with higher scores considered better.Source: Table 13 · p. 18
  • VFM-VAE + BLIP3-o achieves significantly lower gFID on MJHQ-30K at 256 resolution after one pretraining epoch.Lower gFID is the preferred direction for this evaluation.Source: Table 14 · p. 18

Fine-tuning

  • SSIM fine-tuning addresses RGB channel misalignment and edge color noise, while PatchGAN fine-tuning improves supervision for fine reconstruction details.The PatchGAN discriminator supplements the original DINO-based discriminator, whose large patch size provides weak fine-detail supervision.Source: E. Implementation details · p. 18

Diffusion model training

  • Diffusion training with REG changes latent dimensions, patch size, batch size, learning rate, optimizer β2, and attention normalization to stabilize long-term training.Experiments use one node with 8×192GB NVIDIA B200 GPUs, and wall-clock measurements report a speedup over VA-VAE.Source: E. Implementation details · p. 18

F. More qualitative results

The paper supplements its quantitative study with reconstruction comparisons, stage-wise training visualizations, and 256-resolution generation examples. These materials cover VFM-VAE training stages, generative performance, costs, architecture, and training hyperparameters.

Generation examples

Training-stage evaluation

  • VFM-VAE reconstruction and alignment performance across 256-resolution training stages is summarized using CKNNA computed with SigLIP2-Large.Table 15 reports the stage-wise reconstruction and alignment summary.Source: Table 15 · p. 19

Costs and configurations

  • Generative performance and training costs are compared in A100 GPU hours, while the architecture and training hyperparameters are documented separately.Tables 16–18 cover costs, architecture, and training settings.Source: Table 16 · p. 19, Table 17 · p. 19, Table 18 · p. 19

Reconstruction comparisons

Generative training stages

  • Stage-wise visualizations use a fixed random seed and identical initial noise to show generative training behavior and image-generation learning speed.Figure 7 is specifically described as demonstrating accelerated image-generation learning for the proposed approach.Source: F. More qualitative results · p. 18, Figure 7 · p. 20
Loading 2510.18457v3…