PaperlayerView exact revision on arXiv

Source-linked AI summary

GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation

Tianwei Xiong, Jun Hao Liew, Zilong Huang, Jiashi Feng, Xihui Liu

arXiv:2504.08736v2cs.CV

TL;DR

Scaling visual tokenizers can improve reconstruction while degrading downstream AR generation, creating a reconstruction–generation dilemma. GigaTok addresses this with semantic regularization and scaling practices, and reports state-of-the-art reconstruction, AR generation, and representation quality at 3B parameters.

  • Problem

    Scaling visual tokenizers improves reconstruction fidelity but can degrade downstream AR generation, while this scaling problem remains insufficiently studied.

  • Method

    GigaTok uses semantic regularization to align tokenizer features with pre-trained visual representations and combines it with 1D tokenizers, decoder-prioritized scaling, and entropy loss.

  • Results

    GigaTok is the first tokenizer scaled to 3B parameters and achieves state-of-the-art reconstruction, downstream AR generation, and downstream AR representation quality.

  • Takeaways & Limitations

    Constraining latent-space complexity enables tokenizer scaling without the reported reconstruction–generation trade-off, while the 2.9B tokenizer adds about 10% of inference time with a 1.4B AR model.

  • Takeaways & Limitations

    The study focuses on class-conditional image generation; text-conditional image and video generation remain open directions.

Abstract

from arXiv · show

In autoregressive (AR) image generation, visual tokenizers compress images into compact discrete latent tokens, enabling efficient training of downstream autoregressive models for visual generation via next-token prediction. While scaling visual tokenizers improves image reconstruction quality, it often degrades downstream generation quality -- a challenge not adequately addressed in existing literature. To address this, we introduce GigaTok, the first approach to simultaneously improve image reconstruction, generation, and representation learning when scaling visual tokenizers. We identify the growing complexity of latent space as the key factor behind the reconstruction vs. generation dilemma. To mitigate this, we propose semantic regularization, which aligns tokenizer features with semantically consistent features from a pre-trained visual encoder. This constraint prevents excessive latent space complexity during scaling, yielding consistent improvements in both reconstruction and downstream autoregressive generation. Building on semantic regularization, we explore three key practices for scaling tokenizers:(1) using 1D tokenizers for better scalability, (2) prioritizing decoder scaling when expanding both encoder and decoder, and (3) employing entropy loss to stabilize training for billion-scale tokenizers. By scaling to $\bf{3 \space billion}$ parameters, GigaTok achieves state-of-the-art performance in reconstruction, downstream AR generation, and downstream AR representation quality.

1. Introduction

Scaling visual tokenizers improves reconstruction but can degrade downstream autoregressive generation because larger tokenizers produce more complex latent spaces. GigaTok addresses this dilemma with semantic regularization and scaling practices, achieving strong reconstruction, generation, and representation quality at 3B parameters.

  • AR visual generation: Visual tokenizers encode images into discrete tokens, while downstream AR generators learn their distribution through next-token prediction.The tokenizer provides the compact latent space used by the downstream generator.
  • Scaling challenge: Scaling visual tokenizers improves reconstruction fidelity but degrades downstream AR generation, motivating methods that improve both objectives.This reconstruction–generation dilemma is observed in prior work and in Figure 1.
  • Scaling challenge: AR probing shows that larger tokenizers create more complex token distributions that downstream AR models struggle increasingly to learn.The increasing AR generation loss provides evidence for this growing learnability problem.
  • GigaTok: Semantic regularization aligns tokenizer features with pre-trained visual representations to constrain latent-space complexity and support downstream AR modeling.GigaTok uses pre-trained visual models such as DINOv2 for this alignment.
  • GigaTok: GigaTok scales to 3B parameters using 1D tokenizers, decoder-prioritized asymmetric scaling, and entropy loss for billion-scale training.The resulting tokenizer is reported to achieve state-of-the-art reconstruction, downstream AR generation, and downstream AR representation quality.

2. Related Work

Prior work studies image tokenization and semantic or representation alignment, but proper scaling of tokenizer models remains insufficiently explored. GigaTok differs by identifying semantic regularization as central to resolving the reconstruction–generation trade-off.

  • Image tokenizers: Image tokenizers map images into discrete or continuous tokens that downstream generative models can model.Discrete tokenizers predominantly use vector quantization, while newer quantization methods target better codebook scaling.
  • Image tokenizers: Transformer-based tokenizers support convenient scaling, but existing work has not adequately established how to scale tokenizer models effectively.The related-work discussion identifies this as an insufficiently studied problem.
  • Semantic guidance: Representation alignment and semantic guidance have improved visual generative models and tokenizer downstream performance, but GigaTok emphasizes their role in resolving the reconstruction–generation dilemma.The paper distinguishes this focus from existing work.

3. Pilot Study

The pilot study evaluates tokenizer scaling with AR probing, a lightweight proxy for downstream generation. As vanilla tokenizers grow, reconstruction improves while AR generation worsens and token dependencies become harder to learn.

  • AR Probing for Tokenizer Evaluation: AR Probing uses a lightweight downstream AR model to monitor tokenizer effectiveness during training.It is introduced because fully training downstream generators is computationally expensive.
  • AR Probing for Tokenizer Evaluation: The probing model evaluates tokenizer-induced token distributions using gFID, validation loss, and linear probing accuracy.The 111M-parameter model is trained for 50 epochs and is 10× more efficient than the original 343M downstream generator.
  • Naively Scaling Tokenizers Does Not Work: As vanilla 1D tokenizer size increases, reconstruction quality improves while downstream AR Probing gFID consistently degrades.The scaling trend exposes the reconstruction–generation dilemma.
  • Naively Scaling Tokenizers Does Not Work: Increasing AR Probing validation loss indicates that larger tokenizers produce more complicated token dependencies that are harder for AR models to learn.This observation motivates semantic regularization to constrain latent-space complexity.

4. GigaTok

GigaTok is a scalable visual tokenizer built around a hybrid CNN-Transformer architecture, semantic regularization, and training strategies for billion-scale models. It addresses latent-space complexity and convergence challenges that arise as tokenizers scale.

  • Architecture: GigaTok supports 1D and 2D token structures with a hybrid CNN-Transformer tokenizer backbone.The encoder uses CNN blocks followed by Transformer layers and vector quantization; the architecture is designed for scalable tokenizer construction.
  • Scaling Challenge: Scaling tokenizers increases latent-space complexity, making the resulting token distributions harder for downstream autoregressive models to learn.The paper hypothesizes that larger tokenizers capture excessive fine-grained details, producing overly complex latent token dependencies.
  • Semantic Regularization: Semantic regularization aligns tokenizer decoder features with frozen DINOv2 representations to constrain latent-space complexity.The regularization uses cosine similarity between decoder features and projected DINOv2 features, and is added to the VQGAN training objective.
  • Entropy Loss: 2.9B tokenizers fail to converge with the 622M training recipe and exhibit consistently low codebook usage.This motivates adding an entropy penalty to encourage higher codebook utilization during billion-scale training.
  • Entropy Loss: Entropy loss encourages more uniform codebook selection and enables 2.9B tokenizer training to converge properly.The conditional-entropy term reduces quantization uncertainty, while the marginal-entropy term promotes broader codebook usage; with entropy loss, codebook usage quickly reaches a high level.

5. Experiments

Experiments evaluate GigaTok across reconstruction, downstream AR generation, and representation quality, showing that semantic regularization enables tokenizer scaling without the usual reconstruction–generation trade-off. Additional studies examine tokenizer dimensionality, encoder–decoder scaling, system-level performance, inference cost, and regularization strength.

  • Experimental Setup: AR Probing trends correlate with larger AR-model improvements, providing a lower-cost indicator of downstream tokenizer effects.The study uses AR Probing validation loss, gFID, and linear probing to assess token learnability, generation, and representation quality.
  • Scaling with Semantic Regularization: Semantic regularization improves reconstruction fidelity while preventing the validation-loss and gFID degradation seen as vanilla tokenizers scale.The regularized models also show consistently improving downstream representation quality.
  • Scaling with Semantic Regularization: Semantic regularization produces more semantically consistent tokenizer features and reduces latent-space complexity for downstream AR models.PCA visualizations compare features from images in the same golden retriever class.
  • Asymmetric Encoder–Decoder Scaling: Scaling decoders improves reconstruction and downstream generation more than scaling encoders, although encoder scaling still improves downstream models.Table 2 compares S-B with B-S and S-L with B-L under the same training setting.
  • 1D versus 2D Tokenizers: 1D tokenizers achieve better reconstruction and representation quality than 2D tokenizers, with a steeper downstream-generation scaling trend.The authors attribute this scalability advantage to reduced inductive bias.
  • System-level Comparison: A 2.9B GigaTok achieves state-of-the-art reconstruction, while a downstream 1.4B AR model achieves state-of-the-art generation and linear-probing performance on ImageNet.The generation comparison concerns LLM-style autoregressive next-token prediction models.
  • Ablation Study: Choosing the semantic-regularization layer involves a trade-off: stricter regularization improves representation quality but can reduce generation quality.The study selects l = 3 after evaluating this trade-off.

6. Conclusion

The paper identifies increasing latent-space complexity as the source of the reconstruction–generation dilemma and uses semantic regularization plus scaling practices to address it. GigaTok thereby reaches 3B scale with state-of-the-art reconstruction, downstream AR generation, and representation quality.

  • Conclusion: GigaTok addresses tokenizer scaling by regularizing latent-space complexity with pretrained visual representations.The approach aligns tokenizer features with pretrained representations during training.
  • Conclusion: GigaTok is the first 3B tokenizer reported to achieve state-of-the-art reconstruction, downstream AR generation, and downstream AR representation quality.The conclusion attributes these results to semantic regularization together with the explored scaling practices.

A. Limitations and Future Work

The study’s scope is primarily class-conditional image generation, while text-conditional image generation and video generation remain future directions. Other open questions concern resolution transfer for 1D Transformer tokenizers and scaling data and codebook properties.

  • Scope: The study primarily evaluates tokenizer scaling for class-conditional image generation.Extending the approach to text-conditional image generation and video generation remains open.
  • Resolution Generality: 1D Transformer-based tokenizers do not directly support multiple resolutions without additional training adjustments.This differs from CNN-based 2D tokenizers.
  • Future Work: The effects of scaling training data, codebook dimension, and codebook size on downstream autoregressive generation remain unstudied.These factors are identified as future research directions.

C. Detailed GigaTok Implementation

GigaTok uses 1D Q-Former tokenizers and entropy-based regularization to support scalable VQ training. Its entropy loss balances confident quantization with broader codebook utilization, while implementation tricks stabilize hybrid-architecture training.

  • 1D tokenizers: GigaTok builds 1D tokenizers with a Q-Former and initializes queries from multi-level pooled CNN features.The encoder forms a 1D sequence from flattened pooled features, while decoder 2D queries start from the first 1D latent feature.
  • VQ formulation: For VQ tokenizers, quantization probabilities are modeled with a softmax over negative L2 distances between inputs and codebook vectors.The probability of selecting codebook vector c_i is derived from distances between z and all vectors in C.
  • Entropy loss: Entropy loss combines conditional entropy minimization with marginal entropy maximization to improve quantization confidence and codebook utilization.The loss is L_entropy = E_z[H(z_hat|z)] - H(z_hat).
  • Entropy loss: Entropy loss is presented as a detailed VQ formulation rather than as a new contribution.The authors estimate marginal codebook probabilities by averaging conditional probabilities over z.
  • Training stabilization: A shortcut feature-reconstruction path initially stabilizes hybrid training, but one learning-rate warmup epoch with entropy loss can replace it.The shortcut is used during the first 15k iterations in the initial recipe.

D. Full Evaluation Results and Analysis

Full ImageNet evaluations show that tokenizer scaling improves reconstruction quality, while semantic regularization is associated with favorable downstream generation behavior. DINO-based discrimination improves reconstruction metrics but yields less significant generation gains and can destabilize large-tokenizer training.

  • Overall evaluation: Scaling visual tokenizers consistently improves reconstruction quality across multiple metrics on ImageNet 256×256.Table 10 reports tokenizer and downstream AR-model results, with gFID selected from classifier-free-guidance and no-guidance settings.
  • Overall evaluation: Semantic regularization is hypothesized to contribute to the lowest 1.4B AR-model gFID occurring without classifier-free guidance.The same phenomenon is reported for the concurrent FlexTok work despite architectural differences.
  • Discriminator scaling: DINO discriminators improve rFID, but downstream generation-quality improvements are less significant than the reconstruction gains.This comparison uses the same DINO-discriminator configuration as VAR.
  • Discriminator scaling: Adversarial training with DINO discriminators frequently becomes unstable for XL-XXL tokenizers, producing blurry artifacts after the discriminator quickly distinguishes reconstructions.The authors leave discriminator scaling and enhancement strategies for future work.

E. Training Tokenizers for More Iterations

Extending tokenizer training improves reconstruction but eventually harms downstream generation, indicating that training duration introduces a separate scaling dilemma. Linear probing of tokenizer encoders can also misrepresent downstream AR performance, so AR probing is a more direct evaluation for the stated goal.

  • Training duration: Longer tokenizer training consistently improves reconstruction but causes downstream generation quality to improve initially and then degrade.The experiments train five S-S tokenizers for 40 to 120 epochs with a cosine learning-rate scheduler.
  • Training duration: AR-probing validation loss rises with longer training regardless of semantic regularization, indicating increasing latent-space complexity.The authors hypothesize that data scaling may alleviate this issue and leave it for future exploration.
  • Training duration: Allocating compute to model scaling rather than extended training duration may yield better tokenizer performance.This is presented as a practical implication while data scaling remains unexplored.
  • Evaluation metrics: Tokenizer-encoder linear probing can show overfitting even while downstream AR performance continues improving.The discrepancy is observed for the XL-XXL tokenizer across intermediate training checkpoints.
  • Evaluation metrics: AR probing is recommended as a more direct evaluation because the target is representation learning for downstream models rather than tokenizer encoders.The abnormality is especially evident for large tokenizers, where tokenizer linear probing and downstream metrics diverge.
  • Related work: Prior work TiTok leaves the reconstruction-versus-generation scaling dilemma unresolved and therefore trains its best generator on the smallest tokenizer.TiTok explores 1D Transformer tokenizers under high compression but does not solve the scaling trade-off.
  • Related work: ViTok similarly finds asymmetric VAE design beneficial, while GigaTok reports that encoder scaling can also help in its setting.The authors hypothesize semantic regularization may help resolve the dilemma for VAEs but leave that study for future work.
Loading 2504.08736v2…