Source-linked AI summary
Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, Ying Shan
TL;DR
Open-MAGVIT2 tackles limited tokenizer quality and the difficulty of using super-large visual vocabularies in auto-regressive generation. It replicates MAGVIT-v2’s lookup-free tokenizer, factorizes tokens for auto-regressive prediction, and releases scalable models and pretrained variants. The project reports state-of-the-art tokenizer reconstruction and superior plain auto-regressive generation under the evaluated settings.
Problem
Visual generation is limited by tokenizer reconstruction quality, codebook size and utilization, while using a super-large vocabulary in plain auto-regressive models is difficult.
Method
The project replicates MAGVIT-v2’s lookup-free tokenizer and uses asymmetric token factorization with next sub-token prediction for plain auto-regressive visual generation.
Results
The tokenizer achieves state-of-the-art reconstruction on ImageNet and UCF, while the plain auto-regressive model outperforms previous image-generation models using that approach.
Takeaways & Limitations
Open-MAGVIT2 provides open tokenizers and scalable auto-regressive models that demonstrate the potential of super-large codebooks for visual generation.
Takeaways & Limitations
The effectiveness of the 2^18-code codebook may be underestimated by limited data scale and representational capacity sacrificed by token factorization.
Abstract
from arXiv · showhide
The Open-MAGVIT2 project produces an open-source replication of Google's MAGVIT-v2 tokenizer, a tokenizer with a super-large codebook (i.e., $2^{18}$ codes), and achieves the state-of-the-art reconstruction performance on ImageNet and UCF benchmarks. We also provide a tokenizer pre-trained on large-scale data, significantly outperforming Cosmos on zero-shot benchmarks (1.93 vs. 0.78 rFID on ImageNet original resolution). Furthermore, we explore its application in plain auto-regressive models to validate scalability properties, producing a family of auto-regressive image generation models ranging from 300M to 1.5B. To assist auto-regressive models in predicting with a super-large vocabulary, we factorize it into two sub-vocabulary of different sizes by asymmetric token factorization, and further introduce ``next sub-token prediction'' to enhance sub-token interaction for better generation quality. We release all models and codes to foster innovation and creativity in the field of auto-regressive visual generation.
1 INTRODUCTION
Open-MAGVIT2 addresses limited visual-tokenizer performance by replicating MAGVIT-v2’s lookup-free, super-large-codebook tokenizer and extending it to plain auto-regressive generation. The project releases tokenizers and auto-regressive models, including large-scale pretrained variants.
- Motivation: Limited codebook size and utilization constrain existing visual tokenizers, leaving auto-regressive visual generation behind diffusion-based methods.MAGVIT-v2’s Lookup-Free Quantizer is presented as a response to these tokenizer limitations.
- Tokenizer replication: 1.18 versus 1.15 rFID on ImageNet 128×128 shows the open-source replication closely matches MAGVIT-v2, while its pretrained tokenizer surpasses Cosmos and LlamaGen on zero-shot reconstruction.The project also reports state-of-the-art performance on ImageNet and UCF benchmarks and zero-shot reconstruction on COCO and ImageNet.
- Auto-regressive extension: Asymmetric token factorization and next sub-token prediction adapt a super-large vocabulary for plain auto-regressive visual generation.The design targets stronger interaction among sub-tokens while modeling visual sequences with auto-regressive transformers.
- Open-source releases: Open-MAGVIT2 releases ImageNet-trained tokenizers, 300M-to-1.5B auto-regressive models, pretrained tokenizers, and a video tokenizer.The releases span versions V1.0 through V1.2.
2 METHOD
Open-MAGVIT2 uses a visual tokenizer to convert images or videos into discrete representations, then models those tokens auto-regressively. Its lookup-free quantizer enables a highly utilized super-large codebook, while asymmetric factorization and intra-token prediction make the vocabulary tractable for auto-regressive modeling.
- Visual tokenizer: The visual tokenizer encodes inputs into feature maps, quantizes feature vectors into discrete code indices, and decodes those indices back into reconstructed pixels.The encoder uses spatial and temporal downsampling, and decoding maps each code index back to a quantized feature vector.
- Lookup-Free Quantization: Lookup-free quantization represents each codebook entry as a K-bit sign pattern, producing a codebook of 2^K codes without embedding lookup.The quantized representation is obtained by applying the sign function to each latent feature dimension.
- Lookup-Free Quantization: Replacing pairwise code assignment with lookup-free quantization enables a 2^18-code codebook with 100% utilization in the reported experiment.The quantizer also uses entropy-based objectives to encourage confident assignments and simultaneous utilization of the whole codebook.
- Auto-regressive architecture: Asymmetric token factorization splits each LFQ token into subspaces with different vocabulary sizes, embeds them separately, and sums the embeddings as transformer inputs.The design addresses the impracticality of directly optimizing a 2^18-token vocabulary with relatively small auto-regressive transformers.
- Auto-regressive architecture: The auto-regressive transformer models inter-token context and predicts each token’s sub-tokens sequentially to capture intra-token dependencies.Contextual token representations support inter-token modeling, while conditioning sub-token predictions on earlier sub-tokens enhances within-token interaction.
3 EXPERIMENTS
Experiments evaluate Open-MAGVIT2 for image and video reconstruction, zero-shot reconstruction, and class-conditional generation. The tokenizer matches MAGVIT-v2 closely, outperforms prior tokenizers under reported settings, and supports competitive plain autoregressive generation.
- Visual Reconstruction: Open-MAGVIT2 matches MAGVIT-v2 within a 0.03 FID margin on ImageNet 128×128 reconstruction.This comparison incorporates the useful designs proposed in prior MAGVIT-v2 work.
- Visual Reconstruction: Open-MAGVIT2 outperforms previous image tokenizers under fair settings and improves detail, facial, and text reconstruction.The comparison covers ImageNet reconstruction at 256×256 resolution.
- Zero-Shot Reconstruction: The pretrained tokenizer achieves state-of-the-art zero-shot reconstruction on ImageNet and COCO against concurrent methods such as Cosmos.It is pretrained on large-scale image-text datasets for text-conditional image generation.
- Comparison Scope: Residual tokenization methods with better results are omitted because residual techniques are orthogonal and compatible with VQ, LFQ, and FSQ.This defines the comparison scope rather than reporting an additional Open-MAGVIT2 result.
- Video Reconstruction: Open-MAGVIT2 achieves competitive reconstruction among previous video tokenizers on UCF-101.The image tokenizer is extended to video, with reconstruction visualizations provided for UCF-101.
- Visual Generation: Open-MAGVIT2 outperforms all previous image generation models using a plain autoregressive approach on ImageNet.The authors attribute this result to the increased representational capacity of its large codebook.
4 RELATED WORKS
Related work frames visual tokenizers as the discrete representation layer for generation and distinguishes non-autoregressive from autoregressive synthesis. MAGVIT-v2 enlarges the codebook with Lookup-Free Quantization, while autoregressive visual generation remains less advanced than related multimodal understanding.
- Visual Tokenizers: Visual tokenizers map images into compact discrete tokens that generative models consume for sequence modeling.The literature progresses from VQ-VAE through normalized and multi-scale quantization methods.
- Visual Tokenizers: MAGVIT-v2 enlarges the codebook to 2^18 entries using Lookup-Free Quantization to increase individual-token representational capacity.This follows the intuition that reducing code dimension limits the capacity of each token.
- Generation Frameworks: Visual synthesis uses two prevalent frameworks: non-autoregressive generation and autoregressive generation.The distinction concerns how compact discrete image tokens are subsequently generated.
- Non-autoregressive Frameworks: Non-autoregressive methods such as MaskGIT and MAGVIT generate visual tokens in parallel through masked prediction.MAGVIT additionally uses an embedding mask for improved generation quality.
- Autoregressive Frameworks: Autoregressive visual generation has progressed less satisfactorily than autoregressive multimodal visual understanding.Prior approaches include VQGAN’s next-token prediction and VAR’s next-scale prediction.
5 CONCLUSION
The paper presents Open-MAGVIT2 as an open-source tokenizer and autoregressive generation framework built around a super-large codebook. It concludes that larger data and models are needed to fully exploit this design in broader multimodal generation.
- Conclusion: Open-MAGVIT2 re-implements a powerful visual tokenizer and introduces asymmetric token factorization with next sub-token prediction for autoregressive synthesis.The authors report superiority and scalability for plain autoregressive models using this tokenizer.
- Limitations and Future Work: The effectiveness of the 2^18-code super-large codebook may be underestimated by limited data scale and representational loss from token factorization.The authors identify larger datasets and models as directions for future work.
- Limitations and Future Work: Future exploration targets text-conditional image generation, video generation, and models scaled to 7B parameters or larger.The stated goal is to extend Open-MAGVIT2 toward broader multimodal generation applications.