PaperlayerView exact revision on arXiv

Source-linked AI summary

SVG-T2I: Scaling Up Text-to-Image Latent Diffusion Model Without Variational Autoencoder

Minglei Shi, Haolin Wang, Borui Zhang, Wenzhao Zheng, Bohan Zeng, Ziyang Yuan, Xiaoshi Wu, Yuanxing Zhang, Huan Yang, Xintao Wang, Pengfei Wan, Kun Gai, Jie Zhou, Jiwen Lu

arXiv:2512.11749v1cs.CV

TL;DR

Large-scale text-to-image diffusion directly in VFM feature space remains largely unexplored, despite the potential of VFM representations for unifying visual tasks. SVG-T2I scales this approach by training a text-to-image model directly on DINO features. It reports performance comparable to advanced methods, including a GenEval score of 0.74 matching SD3-Medium and surpassing SDXL and DALL-E 2.

  • Problem

    Large-scale, systematic text-to-image diffusion training directly in VFM feature space remains largely absent, and ImageNet class-conditioned results do not establish generalization to realistic text-to-image scenarios.

    Source: §1 · pp. 1, 3

  • Method

    SVG-T2I scales the SVG framework by training a text-to-image diffusion model directly on DINOv3 features, using a single-stream DiT that jointly processes text and image tokens.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    SVG-T2I achieves a GenEval overall score of 0.74, matching SD3-Medium and significantly surpassing SDXL and DALL-E 2, with generative metrics described as comparable to modern advanced methods.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The work identifies VFM semantic space as a potential effective latent manifold for high-resolution synthesis and releases code and model weights to support further research.

    Source: §5 · p. 9

    DINOv2 and DINOv3 representations show resolution-dependent feature instability, which compromises SVG-T2I’s generalization across sizes and its generation quality.

    Source: §5 · p. 10

Abstract

from arXiv · show

Visual generation grounded in Visual Foundation Model (VFM) representations offers a highly promising unified pathway for integrating visual understanding, perception, and generation. Despite this potential, training large-scale text-to-image diffusion models entirely within the VFM representation space remains largely unexplored. To bridge this gap, we scale the SVG (Self-supervised representations for Visual Generation) framework, proposing SVG-T2I to support high-quality text-to-image synthesis directly in the VFM feature domain. By leveraging a standard text-to-image diffusion pipeline, SVG-T2I achieves competitive performance, reaching 0.75 on GenEval and 85.78 on DPG-Bench. This performance validates the intrinsic representational power of VFMs for generative tasks. We fully open-source the project, including the autoencoder and generation model, together with their training, inference, evaluation pipelines, and pre-trained weights, to facilitate further research in representation-driven visual generation.

1 INTRODUCTION

Prior work has incorporated Visual Foundation Model (VFM) features into VAE-based latent diffusion, while direct diffusion in VFM feature space has shown promise for generation and a unified encoder infrastructure. Large-scale text-to-image training in that space remains underexplored, so this paper focuses on that challenge and reports the first large-scale study of it.

1 INTRODUCTION

  • Prior work incorporates pretrained VFM features into VAE-based latent diffusion through feature alignment or joint generation, while SVG and RAE train diffusion directly in high-dimensional VFM feature space.The latter approach is described as improving generation quality and training and inference efficiency, with potential for shared encoder infrastructure across tasks.Source: §1 · p. 1
  • Large-scale, systematic text-to-image studies of VFM-feature diffusion remain largely absent, and class-conditioned ImageNet results cannot establish generalization to realistic text-to-image scenarios.The paper notes ImageNet’s limited scale and category diversity and that most vision encoders are already pretrained on it.Source: §1 · p. 3
  • The paper focuses on whether VFM-derived representations support large-scale, high-resolution text-to-image diffusion, alongside the broader question of a unified feature space for multiple visual tasks.The broader challenge asks whether one feature space can support reconstruction, perception, high-fidelity generation, and semantic understanding without compromising performance.Source: §1 · pp. 1, 3
  • The paper presents the first large-scale validation of text-to-image latent diffusion in VFM feature space and releases its training and inference pipeline with pretrained weights in multiple sizes.The release is intended to facilitate future research.Source: §1 · p. 3

2 RELATED WORKS

Related work spans several generative paradigms and efforts to improve or enrich generative representations. Recent approaches train diffusion directly in VFM features or pair self-distilled VFMs with pixel decoders for reconstruction and generation.

2 RELATED WORKS

  • VAE latent diffusion reduces computational cost while preserving generation quality, but its latent space is described as lacking coherent semantic structure for downstream understanding and perception.The related-work overview also covers GANs, autoregressive models, masked generation, and diffusion as major visual-generation paradigms.Source: §2 · p. 3
  • Representation-focused work regularizes VAE latents, uses diffusion-guided reconstruction, aligns generative latents with semantic embeddings, or incorporates discriminative knowledge into generation.The passage cites these as distinct directions for improving generative representations.Source: §2 · p. 3
  • SVG and RAE train diffusion directly in VFM feature space, while UniLiP and UniFlow pair self-distilled VFMs with pixel decoders for reconstruction and generation.UniLiP and UniFlow use MAR or MetaQueries-style training paradigms.Source: §2 · pp. 3–4
  • The proposed SVG-T2I architecture is illustrated as a single-stream DiT with an optional residual encoder augmenting DINO for reconstruction and transferability.Figure 2 presents the DiT architecture on the left and the optional encoder augmentation on the right.Source: Figure 2 · p. 4

3 METHODOLOGY

SVG-T2I extends flow matching in VFM feature space to large-scale text-to-image generation. Its high-resolution reconstruction findings motivate making the residual encoder optional.

3.1 PRELIMINARIES

  • Flow matching interpolates between data and Gaussian noise, learns a velocity field, and samples by solving the probability-flow ODE.The interpolation is xt = (1 − t)x0 + tϵ for t ∈ [0, 1], with ϵ drawn from a standard Gaussian.Source: §3.1 · p. 4

3.2 SELF-SUPERVISED REPRESENTATIONS FOR VISUAL GENERATION

  • The autoencoder offers a pure configuration using frozen DINOv3 features and a residual configuration whose module compensates for high-frequency details and color-cast artifacts.Both configurations use the same decoder to map features back to pixel space.Source: §3.2 · p. 4
  • SVG-T2I trains a Unified Next-DiT backbone on DINOv3 features, jointly processing text and image tokens as a single sequence with flow matching.The DINOv3-ViT-S/16+ encoder maps an H × W × 3 image to (H/16) × (W/16) × 384 features.Source: §3.2 · p. 5
  • Training first fits the two autoencoders separately from scratch, then trains the DiT with autoencoder-P using a progressive schedule.The residual variant uses reconstruction losses and distribution matching on its residual branch and decoder.Source: §3.2 · p. 5

3.3 SCALING SVG TO HIGHER RESOLUTION

  • High-resolution DINOv3 reconstructions preserve more detail than low-resolution ones, motivating omission of the residual encoder for high-resolution reconstruction or generation.Figure 3 presents reconstructions and feature visualizations at different input resolutions.Source: §3.3 · p. 5, Figure 3 · p. 5
  • Across resolutions, VAE features show stronger scale invariance than DINO features, while exhibiting limited semantic structure.The comparison uses cosine similarity after downsampling higher-resolution features to match lower-resolution spatial size.Source: Figure 4 · p. 6

4 EXPERIMENTS

The experiments evaluate SVG-T2I’s staged training and its text-to-image performance, then analyze cross-resolution feature behavior and remaining generation limitations.

4.1 MODEL TRAINING

  • In the third stage, SVG-T2I transfers learned knowledge to higher resolutions and refines fine-grained visual details using 15M samples; final fine-tuning on 1M high-quality aesthetic samples further enhances realistic, visually appealing outputs.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 0.74 on GenEval matches SD3-Medium and surpasses SDXL and DALL-E 2, while 85.78 on DPG-Bench is comparable to FLUX.1 and HiDream-I1-Full.All evaluation images were generated at 1024 × 1024 resolution.Source: §4.2 · p. 7, Table 5 · p. 8, Table 6 · p. 9

4.3 ANALYSIS

  • VAE features remain nearly resolution-invariant, whereas DINOv3 and DINOv2 features vary substantially across scales.With fixed-size patches, low-resolution inputs yield more compressed, detail-poor features, while high-resolution inputs capture finer local detail; stable cross-resolution behavior is identified as an optimization goal.Source: §4.3 · p. 8
  • SVG-T2I may struggle with highly detailed faces, anatomically accurate fingers, and reliable text rendering.The paper attributes these shortcomings to insufficient training-corpus coverage and the compute demands of modeling fine-grained patterns and precise geometry; specialized data and additional compute may be needed.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

5 CONCLUSION

The paper extends SVG to large-scale, high-resolution text-to-image synthesis and reports competitive generative metrics from VFM representations. It also identifies cross-resolution feature instability in existing VFM encoders as a challenge for generalization.

5 CONCLUSION

  • SVG-T2I extends the SVG framework to large-scale, high-resolution text-to-image synthesis and achieves generative metrics comparable to advanced methods.The authors describe this as validating the feasibility of training a high-quality text-to-image model from scratch using VFM representations.Source: §5 · p. 9
  • The authors open-source the training, inference, and evaluation code and model weights to support further research and reproducibility.Source: §5 · p. 9
  • DINOv2 and DINOv3 representations vary across input resolutions, compromising generalization across image sizes and generation quality.The paper identifies scale-invariance as a focus for future research.Source: §5 · p. 10

A MORE QUALITATIVE RESULTS

The supplementary qualitative results show SVG-T2I outputs at 1080 × 1080 resolution, highlighting their diversity and visual quality.

A MORE QUALITATIVE RESULTS

B PROMPTS USED IN FIGURE 1

The appendix provides prompts used for Figure 1, spanning portrait, food, wildlife, botanical, and landscape images. The prompts specify subjects and often add scene, lighting, and composition details.

B PROMPTS USED IN FIGURE 1

  • The listed prompts and instructions can be entered into the corresponding model to reproduce the Figure 1 generation results.Some prompt entries are split across passages, including the cat description and mountain-lake prompt.Source: B PROMPTS USED IN FIGURE 1 · pp. 17–19
  • Portrait prompts describe a woman in an embroidered hat and fur-collared coat, and a woman in a lavender gown with a jeweled headpiece.They also include a formal black-and-white portrait of a man.Source: B PROMPTS USED IN FIGURE 1 · pp. 17, 19, 20
  • Food and still-life prompts depict fruit salad, Dalgona coffee, wine glasses with grapes and bread, and sprinkled cupcakes.The wine prompt describes the glasses and surrounding props in detail.Source: B PROMPTS USED IN FIGURE 1 · pp. 17, 18, 19, 21
  • Other prompts cover a Bengal tiger, a Bengal cat, cosmos flowers, a rose, tulips, and a snow-covered mountain lake.Several pair the subject with detailed setting, lighting, or composition descriptions.Source: B PROMPTS USED IN FIGURE 1 · pp. 18–21

Related papers

Loading 2512.11749v1…