PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Generation Tuning

Jiahao Guo, Sinan Du, Jingfeng Yao, Wenyu Liu, Bo Li, Haoxiang Cao, Kun Gai, Chun Yuan, Kai Wu, Xinggang Wang

arXiv:2511.23469v1cs.CV

TL;DR

Whether pretrained VLM visual representations can support generation remains underexplored, despite their alignment with language. VGT aligns VLM semantic encoders with pixel decoders and reports strong reconstruction and autoregressive generation results, alongside faster convergence.

  • Problem

    It remains underexplored whether VLM representations optimized for multimodal understanding can support visual generation, while pixel-level VAE latents are poorly aligned with autoregressive modeling.

    Source: §1 · p. 2

  • Method

    VGT aligns pretrained VLM semantic encoders with pixel decoders through VGT-AE, then tunes the model to predict semantically aligned latents for continuous autoregressive generation.

    Source: §1 · p. 2

  • Results

    VGT reports 26.67 PSNR and 0.50 rFID for reconstruction, 0.77 GenEval and 78.73 DPG-Bench for generation, and a 20x speedup.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VGT-AE’s alignment of pretrained VLM semantic encoders with lightweight pixel decoders addresses the VAE–autoregressive mismatch and supports stable, efficient continuous-latent learning.

    Source: §5 · p. 9

    The first-stage latent distribution does not conform to the standard Gaussian prior required for effective flow-based generation, challenging unregularized latent spaces.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Large Vision Language Models (VLMs) effectively bridge the modality gap through extensive pretraining, acquiring sophisticated visual representations aligned with language. However, it remains underexplored whether these representations, optimized for multimodal understanding tasks, harbor an inherent potential for visual generation. In this paper, we propose VGT, Visual Generation Tuning, a novel paradigm designed to stimulate the underlying capabilities of visual generation within any vision language models. By performing efficient visual generation tuning on well-pretrained VLMs, we significantly mitigate the alignment costs and accelerate the convergence of autoregressive modeling in the continuous space (20x speedup). Specifically, we dismiss the entangled pixel-level VAEs designed for diffusion transformers and formulate VGT-AE through aligning the semantic encoders from pretrained VLMs with the latent representations of pixel decoders. In image reconstruction tasks, we achieve 26.67 PSNR and 0.50 rFID at a 28x compression ratio, outperforming specialized VAEs; in visual generation tasks, we achieve state-of-the-art outcomes among autoregressive models, 0.77 on GenEval and 78.73 on DPG-Bench. Furthermore, our proposed VGT showcases significant scaling promise and is versatile for endowing any VLMs trained for multimodal understanding with the capabilities of visual generation, which paves the new avenue to explore next-generation unified multimodal foundation models. Models and codes are available at https://github.com/hustvl/VGT.

1. Introduction

The introduction identifies a mismatch between pixel-focused VAE latents and autoregressive generation, then asks whether pretrained VLM representations can support visual generation. It proposes VGT and reports reconstruction, generation, and convergence results.

1. Introduction

  • Earlier autoregressive visual-generation methods used VQ codebooks that introduce quantization error, while MAR predicts continuous VAE latents with a lightweight flow-matching head.Source: §1 · pp. 1–2
  • Pixel-optimized vanilla VAE latents lack semantic structure, which is associated with training instability and frequent variance collapse in autoregressive generation.The introduction asks whether VLMs’ alignment with discriminative visual representations can be used to support models that combine multimodal understanding and visual generation.Source: §1 · p. 2
  • VGT uses pretrained VLM semantic encoders in VGT-AE, then predicts semantically aligned image latents with a lightweight flow-matching head for continuous autoregressive generation.VGT-AE uses two-stage training, including self-distillation to preserve latent semantic structure, followed by channel normalization and noise regularization.Source: §1 · p. 2
  • 26.67 PSNR and 0.50 rFID for reconstruction, 0.77 GenEval and 78.73 DPG-Bench for generation, and a 20x speedup are reported for VGT.The paper characterizes its reconstruction and generation performance as state of the art.Source: §1 · p. 2

2. Related Work

Related work covers discrete and continuous visual tokenizers, then reviews autoregressive generation strategies and the mismatch between VAE latents and autoregressive modeling. VGT aligns VLM semantic features with compact latents for continuous autoregressive generation.

2.1. Visual Tokenizers for Generative Modeling

  • VQ-VAE [53] and VQGAN [11] exemplify discrete tokenizers, while later approaches improve codebook use and semantic alignment but retain reconstruction-fidelity versus semantic-comprehension trade-offs.Source: §2.1 · p. 2
  • Continuous tokenizers use VAEs for diffusion, and semantic-feature approaches such as REPA [64], REPA-E [25], RAE [67], ClipGen [56], and SVG [42] remain high-dimensional or diffusion-oriented, limiting efficient autoregressive integration.The paper positions its approach as aligning VLM semantic encoders with low-dimensional VAE latents for continuous autoregressive generation.Source: §2.1 · p. 2

2.2. Autoregressive visual generation

  • Autoregressive visual generation spans next-scale, next-set, and next-token prediction, with parallel prediction and iterative refinement used by next-set methods such as MaskGIT [4] and MAR [26].Source: §2.2 · p. 2
  • Figure 2 depicts VGT-AE aligning a VLM’s semantic encoder through reconstruction training, followed by a lightweight flow-matching head that predicts aligned image latents.The figure presents this as a route to continuous autoregressive image generation.Source: Figure 2 · p. 3
  • Continuous next-token methods use sequential image generation, but borrowing pixel-level VAEs designed for diffusion overlooks their misalignment with autoregressive modeling.Source: §2.2 · p. 3

3. Method

VGT-AE is motivated by the misalignment between vanilla VAE latents and continuous autoregressive modeling, and is trained to balance semantic structure, reconstruction, and autoregressive compatibility. QueryAR uses position queries to support flexible generation order in a standard autoregressive framework.

3.1. Motivation

  • Self-distillation that preserves semantic structure can enhance robustness in continuous autoregressive generation, whereas over-optimizing reconstruction degrades generative capability.The pilot experiments extend this trade-off to continuous autoregressive modeling.Source: Table 1 · p. 4

3.2. VGT-AE

  • VGT-AE uses a two-stage training paradigm to optimize reconstruction quality and autoregressive compatibility while addressing the limitations of existing visual tokenizers.Its architecture combines a pretrained VLM semantic encoder, a residual projection that compresses features to dimension 32, and a DC-AE-adapted pixel decoder.Source: §3.2 · p. 4
  • Stage one combines pixel reconstruction with semantic self-distillation, achieving high-fidelity reconstruction while distilling semantic structure into the compact latent space.The stated loss also includes perceptual and adversarial terms, and uses a frozen pretrained encoder as the distillation target.Source: §3.2 · p. 4
  • Stage-one latents do not conform to the zero-mean, unit-variance Gaussian prior required for effective flow-based generation, motivating latent-space regularization.Stage two freezes the encoder and optimizes the decoder and projection module with channel-wise normalization and Gaussian noise; the passage gives noise standard deviation 0.1.Source: §3.2 · p. 4

3.3. Autoregressive Modeling

  • QueryAR interleaves learnable position queries with latents in a randomly permuted sequence, enabling flexible generation orders while retaining autoregressive properties.The causal context contains previously generated latents and their associated position queries.Source: §3.3 · p. 4
  • QueryAR supports partially parallel decoding by processing target position queries for multiple subsequent positions in one language-model forward pass.The resulting hidden states condition the flow-matching head for deterministic sampling.Source: §3.3 · p. 5

4.1. Experimental Setups

The experiments use open-source image datasets for pretraining and high-quality aesthetic samples for supervised fine-tuning. The setup specifies VLM families, model components, training parameters, and evaluation benchmarks.

4.1. Experimental Setups

  • Pretraining uses 27M BLIP3-o samples recaptioned by Qwen2.5-VL-7B, 5M CC12M samples, and 4M JourneyDB images; supervised fine-tuning uses 200K aesthetic samples.The fine-tuning samples come from ShareGPT-4o, Echo-4o, and BLIP3-o.Source: §4.1 · p. 5
  • The 0.6B- and 1.6B-parameter models use Qwen2.5-VL or InternVL3 vision encoders, DC-AE decoders, and QueryAR with the underlying VLM language model.Qwen2.5-VL models are pruned to reach the target sizes.Source: §4.1 · p. 5
  • Pretraining runs for 100K steps with AdamW, batch size 256, and learning rate 2.0 × 10−4; generation fine-tuning runs for 5,000 iterations at 5 × 10−5.A causal mask limits sequences to 1024 tokens; ablations run for 50K steps.Source: §4.1 · p. 5
  • Reconstruction is evaluated with rFID, PSNR, and SSIM on ImageNet-1K; generation uses GenEval and DPG-Bench, while multimodal understanding uses MME-P, MMB, TQA, and AI2D.The passage identifies MME-Perception, MMBench, and TextVQA as the expansions of MME-P, MMB, and TQA.Source: §4.1 · p. 5

4.2. Visual Tokenizer

VGT-AE uses a two-stage training strategy to align VLM vision encoders with high-fidelity image reconstruction. The section reports strong reconstruction metrics and includes a qualitative input–reconstruction visualization.

4.2. Visual Tokenizer

  • On ImageNet 256×256, VGT-AE-InternViT achieves 0.50 rFID, 26.67 PSNR, and 0.863 SSIM, surpassing existing generative-only and CLIP-based tokenizers.The tokenizer bridges VLM vision-encoder representations with high-fidelity reconstruction through two-stage training.Source: §4.2 · p. 5
  • Table 3 compares visual-generation quality on GenEval and DPG-Bench; its caption reports VGT-Qwen2.5-VL as state of the art on GenEval with 25M training samples.The caption describes the VGT models as highly competitive overall.Source: Table 3 · p. 6
  • Figure 4 presents the input image alongside its VGT-AE-InternViT reconstruction.The input is on the left and the reconstruction is on the right.Source: Figure 4 · p. 6

4.3. Visual Generation

VGT produces diverse, realistic images from pretrained vision–language models, with strong benchmark performance and results on constraint-based tasks. The reported comparisons highlight competitive generation using 25M training samples.

4.3. Visual Generation

  • 0.77 overall on GenEval, VGT-Qwen2.5-VL surpasses Janus-Pro [9], TokenFlow [36], and SimpleAR [54], and competes favorably with or exceeds diffusion models across sub-metrics.The compared diffusion models include SDXL, SD3-Medium [40], and DALL-E 3 [38].Source: §4.3 · p. 6
  • VGT-AE’s VLM-based representations provide inherent vision–language alignment, supporting semantic consistency with textual instructions and avoiding structural information loss common in traditional VAEs.Figure 5 depicts VGT models producing diverse and realistic high-quality images.Source: §4.3 · p. 6, Figure 5 · p. 6
  • VGT shows clear advantages on constraint-based Count and Position tasks; VGT-0.6B also surpasses NextStep-1 [50] and Janus-Pro [9] on several metrics.The paper contrasts this with conventional autoregressive models that typically use hundreds of millions to billions of training samples and can show unstable performance.Source: §4.3 · pp. 6–7
  • VGT-Qwen2.5-VL-1.6B attains GenEval performance comparable to SD3-Medium [40] using only 25M training samples, while SD3-Medium uses 2B parameters and expensive training procedures.The authors characterize VGT as more training-efficient than diffusion-based approaches.Source: §4.3 · p. 7

4.4. Ablation Studies

The ablations examine five design choices, finding that decoder architecture, latent regularization, representation structure, component matching, and parallel decoding shape VGT performance.

4.4.1. VGT-AE Architecture

  • With InternVL3, DC-AE lowers rFID from 15.80 with ViT to 1.13 and reaches 23.95 PSNR; DC-AE and SD-VAE outperform ViT across encoders.DC-AE is selected as the default for balancing semantic structure, detail restoration, and parameter efficiency.Source: §4.4.1 · p. 7, Table 4 · p. 7

4.4.2. VGT-AE Training Methodology

  • Stage 1 reaches GenEval=0.36, while adding normalization and mild noise in Stage 2 raises it to 0.54 despite slightly worse reconstruction.Reconstruction quality decreases as noise increases, while generation peaks at moderate noise levels of σ = 0.4–0.6.Source: §4.4.2 · p. 7, Figure 6 · p. 7, Table 5 · p. 7

4.4.3. Reconstruction vs. Generation

  • The reconstruction-oriented InternVL3 variant scores best on reconstruction but worse on GenEval and DPG-Bench; relaxing reconstruction improves generation as latent manifolds become more dispersed and semantically separated.Despite lower automatic generation metrics, InternVL3 produces sharper illumination and finer textures than Qwen2.5-VL.Source: §4.4.3 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8

4.4.4. Different Language Models in VGT

  • Matched AE–LLM pairs perform best overall, while a mismatched Qwen2.5-VL AE and InternVL3 LLM still surpass the DC-AE VAE baseline on generation metrics.The results support transferring VLM representations to autoregressive generation with low alignment cost.Source: §4.4.4 · pp. 8–9, Table 7 · p. 8

4.4.5. QueryAR vs. MAR

  • At 4× acceleration, QueryAR achieves 0.59 GenEval and 78.14 DPG-Bench, outperforming MAR, whose GenEval falls from 0.51 at 1× to 0.46 at 4×.Figure 9 reports high-quality outputs at 16× acceleration, with critical visual details and semantic consistency preserved.Source: §4.4.5 · p. 9, Figure 9 · p. 9, Table 8 · p. 9

5. Conclusion

VGT aligns pretrained VLM semantic encoders with pixel decoders to support reconstruction and autoregressive visual generation. Experiments report strong reconstruction and generation performance, fast convergence, and scalability.

5. Conclusion

  • VGT-AE aligns pretrained VLM semantic encoders with lightweight pixel decoders to support stable, efficient learning in a continuous latent space.The approach is presented as a general blueprint for adding visual generation to multimodal understanding models.Source: §5 · p. 9
  • VGT-AE achieves state-of-the-art reconstruction under high compression, while VGT advances autoregressive visual generation with competitive GenEval and DPG-Bench scores.The conclusion also reports fast convergence and strong scalability.Source: §5 · p. 9
  • VGT offers a path toward unified multimodal foundation models combining perception and generation.Source: §5 · p. 9
Loading 2511.23469v1…