Towards Scalable Pre-training of Visual Tokenizers for Generation

Jingfeng Yao; Yuda Song; Yucong Zhou; Xinggang Wang

Source-linked AI summary

Towards Scalable Pre-training of Visual Tokenizers for Generation

Jingfeng Yao, Yuda Song, Yucong Zhou, Xinggang Wang

arXiv:2512.13687v2Dec 15, 2025cs.CV

TL;DR

Reconstruction-only visual tokenizer pre-training can favor low-level information, leaving the scaling of tokenizer compute poorly connected to generation quality. VTP jointly trains contrastive, self-supervised, and reconstruction objectives, and reports stronger generation, understanding, and scaling as tokenizer compute, parameters, and data increase.

Problem
Reconstruction-only training can bias visual tokenizer latents toward low-level information, while systematic scaling for downstream generation remains underexplored.
Method
VTP jointly optimizes image-text contrastive, self-supervised, and reconstruction objectives in a visual tokenizer pre-training framework.
Results
VTP shows steadily improving generative performance with scaled tokenizer compute, parameters, and data, including 65.8% FID improvement from 10× more compute.
Takeaways & Limitations
Semantic understanding is a key driver of generation, and perception-oriented tokenizer pre-training avoids the early saturation reported for reconstruction-only tokenizers.
Takeaways & Limitations
Representation-based prior methods can be limited by existing foundation models, causing a low performance ceiling or substantial reconstruction loss.

Abstract

from arXiv · show

The quality of the latent space in visual tokenizers (e.g., VAEs) is crucial for modern generative models. However, the standard reconstruction-based training paradigm produces a latent space that is biased towards low-level information, leading to a foundation flaw: better pixel-level accuracy does not lead to higher-quality generation. This implies that pouring extensive compute into visual tokenizer pre-training translates poorly to improved performance in generation. We identify this as the ``pre-training scaling problem`` and suggest a necessary shift: to be effective for generation, a latent space must concisely represent high-level semantics. We present VTP, a unified visual tokenizer pre-training framework, pioneering the joint optimization of image-text contrastive, self-supervised, and reconstruction losses. Our large-scale study reveals two principal findings: (1) understanding is a key driver of generation, and (2) much better scaling properties, where generative performance scales effectively with compute, parameters, and data allocated to the pretraining of the visual tokenizer. After large-scale pre-training, our tokenizer delivers a competitive profile (78.2 zero-shot accuracy and 0.36 rFID on ImageNet) and 4.1 times faster convergence on generation compared to advanced distillation methods. More importantly, it scales effectively: without modifying standard DiT training specs, solely investing more FLOPS in pretraining VTP achieves 65.8\% FID improvement in downstream generation, while conventional autoencoder stagnates very early at 1/10 FLOPS. Our pre-trained models are available at https://github.com/MiniMax-AI/VTP.

1. Introduction

The paper argues that reconstruction-only tokenizer pre-training can favor low-level information, limiting generation, and introduces VTP to learn semantically structured latent spaces. Experiments show that perception-oriented pre-training improves generation and scales with compute, parameters, and data.

1. Introduction: Reconstruction quality alone does not guarantee better generation because reconstruction-only training can bias latent spaces toward low-level information.The paper describes a trade-off in which scaling reconstruction training may compromise generation.
1. Introduction: VTP jointly optimizes image-text contrastive, self-supervised, and reconstruction objectives to capture semantic, spatial, and pixel-level information.The framework uses a ViT-based autoencoder and combines cross-modal alignment with self-distillation and masked image modeling.
1. Introduction: Understanding correlates positively with generation quality, indicating that semantically meaningful latent representations accompany stronger generative performance.The paper reports this relationship during visual tokenizer pre-training.
1. Introduction: 65.8% FID improvement follows 10× more tokenizer pre-training compute, while conventional autoencoders saturate early.The diffusion model’s training configuration remains fixed in this comparison.
1. Introduction: VTP’s final tokenizer reaches 0.36 rFID, 78.2% zero-shot accuracy, 85.7% linear probing accuracy, and 1.11 gFID on ImageNet.The reported profile combines reconstruction, understanding, and generation metrics.

2. Related Work

Related work studies how to structure visual tokenizer latents with pre-trained representations, but systematic scaling of first-stage tokenizer pre-training for downstream generation remains underexplored. Existing approaches can also inherit performance ceilings or reconstruction costs from their foundation models.

2. Related Work: Earlier self-supervised pre-training combines self-distillation and masked image modeling to learn generalized representations.DINO, iBOT, and DINOv2 exemplify this pre-training direction.
2. Related Work: Systematic study of how to pre-train LDM visual tokenizers for stronger second-stage generation remains limited.The paper identifies this gap within LDM’s decoupled tokenizer-and-generator framework.
2. Related Work: Representation-based methods use distillation or separated semantic and pixel-level spaces to improve latent structure and generation.Examples include VA-VAE, ImageFolder, and MAETok.
2. Related Work: Existing methods can face a low performance ceiling or substantial reconstruction loss because they depend on current foundational models.The cited limitation applies to approaches that structure latents using pre-trained representations.

3. Visual Tokenizer Pre-training

VTP combines reconstruction with representation-learning objectives in a Vision Transformer autoencoder to learn visual tokenizers that are semantically rich, accurately reconstructive, and generation-friendly.

3. Visual Tokenizer Pre-training: VTP integrates representation learning with conventional reconstruction to learn semantically rich, accurate, and generation-friendly visual representations.The framework combines cross-modal alignment, self-supervised learning, and pixel reconstruction objectives.
3. Visual Tokenizer Pre-training: The tokenizer uses a fully Vision Transformer architecture with a bottleneck mapping images into a d-dimensional latent space for downstream objectives.Encoder features support the text encoder, EMA teacher, and pixel decoder.
3.2. Visual Reconstruction: Reconstruction pre-training minimizes combined L1 and perceptual losses, while a later decoder-only GAN fine-tuning stage improves image fidelity.The tokenizer is frozen during the second-stage pixel-decoder fine-tuning.
3.3. Self-Supervised Learning: Self-supervised learning combines masked image modeling and self-distillation across global and local augmented views.The visual tokenizer processes masked or augmented views while an EMA teacher supplies corresponding targets.
3.5. Overall Objective: The overall objective is a weighted combination of reconstruction, self-supervised, and image-text contrastive losses, with separate batch sampling accommodating their different batch-size needs.Contrastive learning uses corresponding image-text pairs as positives and non-corresponding pairs as negatives.

4. Experiments

The experiments show that reconstruction-only pre-training improves pixel fidelity but fails to scale downstream generation, whereas representation-aware VTP improves generation, understanding, and scalability together.

Scaling up Visual Tokenizer Pre-training: rFID improves from 2.0 to 0.5 while gFID worsens from 55.04 to 58.56 when reconstruction-only training scales.This scaling paradox indicates that better reconstruction does not translate into better generation.
Scaling up Visual Tokenizer Pre-training: Representation-augmented training enables stable, simultaneous improvement in understanding and reconstruction, unlike reconstruction-only autoencoders.Both CLIP+AE and SSL+AE exhibit this behavior as computational budgets increase.
Scaling up Visual Tokenizer Pre-training: Understanding tasks emerge as the dominant factor for generation, with joint representation and reconstruction objectives improving reconstruction, understanding, and generation concurrently.Pure reconstruction instead produces stagnation in understanding and generation despite improved reconstruction.
Scaling Properties with Parameters: gFID improves from 31.28 to 26.12 with larger encoder models and from 26.12 to 24.08 after pixel-decoder expansion, while baseline AE remains near 57.These results establish parameter scalability for VTP under the reported experiments.
Scaling Properties with Data: VTP FID improves from 47.59 to 27.45 as training data grows, whereas AE changes only from 58.37 to 56.71 under identical downstream DiT FLOPs.VTP consistently outperforms the conventional autoencoder across all evaluated data scales.

5. Scaling to Text-to-Image Generation

VTP’s scaling properties extend to text-to-image generation on LAION, where semantic-aware pre-training improves convergence and continued compute scaling improves downstream performance.

5. Scaling to Text-to-Image Generation: Semantic-aware tokenizer pre-training achieves significantly faster convergence than reconstruction-only autoencoding for downstream T2I generation.The evaluation uses LAION text-to-image generation with a DiT-XL downstream model.
5. Scaling to Text-to-Image Generation: Increasing tokenizer pre-training compute continues to improve downstream T2I generation, confirming that VTP’s scaling properties generalize beyond ImageNet.The study evaluates VTP tokenizers across different pre-training compute budgets.
5. Scaling to Text-to-Image Generation: Adding diverse perception losses produces steady gains in downstream T2I generation quality.The ablation varies the variety of perception losses incorporated during VTP pre-training.
5. Scaling to Text-to-Image Generation: CLIP loss substantially improves text rendering in text-to-image generation.The CLIP-loss tokenizer shows a significant text-rendering advantage in the ablation.

6. Further Scaling and Comparison

Further scaling experiments show that VTP improves generation as tokenizer size and compute increase, outperforming fixed-representation alternatives without changing standard DiT training specifications.

6. Further Scaling and Comparison: VTP’s downstream generation improves consistently from the S to L tokenizer scale, whereas RAE performance degrades at larger scales.The comparison uses identical DiT training configurations and is summarized in the scalability comparison table.
6. Further Scaling and Comparison: VTP preserves fine-grained details better than RAE because reconstruction remains part of its training.The comparison is made against the fixed-representation RAE approach.
6. Further Scaling and Comparison: 1.11 gFID with guidance makes VTP-L superior to all prior methods under the standard LightningDiT architecture.The reported comparison is on ImageNet generation; VTP also reaches 2.60 and 2.03 gFID without guidance in 80 epochs.
6. Further Scaling and Comparison: Figure 9 visualizes reconstruction and generation for the section’s qualitative comparison.The figure is identified as covering both reconstruction and generation visualizations.

7. Conclusion

The paper concludes that perception-oriented visual tokenizer pre-training makes semantic understanding central to generation and enables generative performance to scale with compute, model size, and data.

7. Conclusion: Perception-oriented tokenizer pre-training unlocks scalable generative performance by making semantic understanding a key driver of generation.The conclusion contrasts this behavior with traditional reconstruction-only tokenizers, whose performance saturates at small scale.
7. Conclusion: Unlike reconstruction-only tokenizers, VTP continues gaining generative performance as compute budget, model size, and data scale increase.The conclusion presents this as a new scaling law for generation.