Source-linked AI summary
BitDance: Scaling Autoregressive Generative Models with Binary Tokens
Yuang Ai, Jiaming Han, Shaobin Zhuang, Weijia Mao, Xuefeng Hu, Ziyan Yang, Zhenheng Yang, Yali Wang, Huaibo Huang, Xiangyu Yue, Hao Chen
TL;DR
Visual autoregressive generation must balance expressive, stable tokens with tractable and efficient sampling. BitDance uses high-entropy binary tokens, binary diffusion, and next-patch diffusion, achieving strong generation quality and substantial speedups across image-generation settings.
Problem
Visual AR models struggle to combine expressive tokens, stable long-sequence generation, tractable sampling over expansive vocabularies, and efficient sequential inference.
Method
BitDance combines a large-vocabulary binary tokenizer with a binary diffusion head and next-patch diffusion for joint multi-token prediction.
Results
BitDance achieves superior class-conditional and text-to-image generation, including FID 1.24 on ImageNet 256×256, 8.7× speedup with 260M parameters, and over 30× speedup at 1024×1024.
Takeaways & Limitations
High-entropy binary representations and joint diffusion-based prediction support scalable, precise, and efficient autoregressive visual generation.
Takeaways & Limitations
Bit-wise independent sampling fails to capture inter-bit correlations, degrading sampling precision and generative quality.
Abstract
from arXiv · showhide
We present BitDance, a scalable autoregressive (AR) image generator that predicts binary visual tokens instead of codebook indices. With high-entropy binary latents, BitDance lets each token represent up to $2^{256}$ states, yielding a compact yet highly expressive discrete representation. Sampling from such a huge token space is difficult with standard classification. To resolve this, BitDance uses a binary diffusion head: instead of predicting an index with softmax, it employs continuous-space diffusion to generate the binary tokens. Furthermore, we propose next-patch diffusion, a new decoding method that predicts multiple tokens in parallel with high accuracy, greatly speeding up inference. On ImageNet 256x256, BitDance achieves an FID of 1.24, the best among AR models. With next-patch diffusion, BitDance beats state-of-the-art parallel AR models that use 1.4B parameters, while using 5.4x fewer parameters (260M) and achieving 8.7x speedup. For text-to-image generation, BitDance trains on large-scale multimodal tokens and generates high-resolution, photorealistic images efficiently, showing strong performance and favorable scaling. When generating 1024x1024 images, BitDance achieves a speedup of over 30x compared to prior AR models. We release code and models to facilitate further research on AR foundation models. Code and models are available at: https://github.com/shallowdream204/BitDance.
1 Introduction
BitDance addresses visual autoregressive generation’s token-design, sampling, and inference bottlenecks with high-entropy binary tokens, binary diffusion, and next-patch diffusion. It reports strong quality and speed across class-conditional and text-to-image generation.
- 1 Introduction: BitDance combines a large-vocabulary binary tokenizer, binary diffusion head, and next-patch diffusion for scalable autoregressive image generation.The components target representation quality, sampling in expansive discrete spaces, and efficient multi-token prediction.
- 1 Introduction: Binary representations scale vocabulary size up to 2^256, surpass continuous VAEs in reconstruction fidelity, and help regularize long-sequence generation.The compact binary latent space preserves fine-grained visual details while mitigating error accumulation.
- 1 Introduction: Binary diffusion jointly models binary channels in continuous hypercube space, avoiding conventional classifiers’ parameter-efficiency versus sampling-accuracy trade-off.Direct categorical modeling requires h × 2^d parameters, while bit-wise independence sacrifices inter-bit correlations and sampling fidelity.
- 1 Introduction: Next-patch diffusion jointly predicts highly correlated local tokens, enabling precise parallel prediction instead of independently factorized token sampling.The method extends the binary diffusion head to model multiple visual tokens together.
- 1 Introduction: On ImageNet 256×256, BitDance achieves FID 1.24; a 260M-parameter model outperforms a 1.4B parallel AR model with 8.7× speedup.For 1024×1024 text-to-image generation, it achieves over 30× speedup versus standard next-token AR models and reports strong benchmark scores.
- 1 Introduction: Experiments across class-conditional and text-to-image generation show superior generative quality, efficient inference, and favorable scaling.The reported results cover both visual generation settings and high-resolution synthesis.
2 Related Work
Related work contrasts continuous and discrete visual representations and develops alternatives to standard raster-scan autoregressive generation. Prior acceleration methods use masking, scale prediction, grouping, or randomized ordering, but leave further challenges for efficient generation.
- 2 Related Work: VAEs provide high-fidelity continuous latents, whereas VQ tokenizers can suffer quantization errors, codebook instability, and degraded reconstruction.This motivates exploring larger discrete vocabularies with higher token entropy.
- 2 Related Work: Standard visual AR models quantize images into discrete tokens and predict them in raster-scan order, while newer work explores continuous spaces, randomized ordering, and alternative modeling primitives.Examples include token-level diffusion heads and continuous AR frameworks for high-fidelity synthesis.
- 2 Related Work: Acceleration approaches include MAE-style masking, next-scale prediction, grouping, and random-order modeling for parallel or arbitrary-position token prediction.These methods include Mask-GIT, MAR, VAR, PAR, RandAR, and ARPG.
3 BitDance
BitDance combines a high-entropy binary tokenizer with binary diffusion heads to make expressive visual tokens sampleable and enable parallel next-patch generation. Its patch-wise autoregressive design models local token interactions while reducing inference cost.
- Binary Visual Tokenizer and Sampling: Group-wise LFQ partitions d channels into g groups, enabling codebook-size scaling to 2^256 while balancing computational efficiency and optimization accuracy.The resulting discrete tokenizer achieves reconstruction fidelity comparable to continuous VAEs.
- Binary Visual Tokenizer and Sampling: 2^d possible indices make conventional classification prohibitively expensive for d-bit tokens, reaching approximately 4.4 trillion parameters when h = 1024 and d = 32.Bit-wise independence reduces this overhead but fails to capture inter-bit correlations, degrading sampling precision and generative quality.
- Next-Patch Diffusion: The architecture flattens binary latents in patch-wise raster-scan order and uses a binary diffusion head for efficient, precise parallel prediction.This design extends the autoregressive sequence from individual tokens to spatial patches.
- Binary Diffusion Head: BitDance represents binary tokens as continuous hypercube vertices and uses a diffusion objective to jointly model their binary channels.At inference, it starts from Gaussian noise, integrates a learned velocity field with an Euler solver, and hard-binarizes the result.
- Next-Patch Diffusion: Next-patch diffusion predicts p × p tokens jointly, exploiting strong local spatial dependencies instead of sampling every token independently.A block-wise causal mask makes tokens within each patch mutually visible while preserving autoregressive dependencies across patches.
4.1 Scaling up Token Entropy
BitDance scales binary token entropy to improve visual reconstruction and downstream generative quality. Larger models better exploit the expanded vocabulary, while small models struggle with convergence.
- The tokenizer study compares 16× and 32× downsampling configurations with codebook sizes of 2^32 and 2^128 or 2^256.Training uses DataComp-1B supplemented with face and text datasets at 256×256 resolution.
- Token entropy scaling markedly improves discrete reconstruction accuracy, narrowing the gap with continuous models.A 16× tokenizer with a 2^32 codebook outperforms the continuous SD-VAE.
- Larger autoregressive Transformers effectively leverage large vocabularies to achieve superior ImageNet generative quality.Small-scale Transformers struggle to converge when the vocabulary is large.
4.2 Class-conditional Image Generation
On ImageNet 256×256, BitDance improves class-conditional generation as model size grows and remains strong in parallel decoding. Its 1B model reaches FID 1.24, while a 260M model surpasses a 1.4B parallel baseline.
- The ImageNet evaluation uses FID as the main metric, with Inception Score, Precision, and Recall as complementary measures.
- The class-conditional benchmark compares BitDance with parallel methods using throughput measured on one A100 at batch size 64.
- Performance consistently improves as BitDance model size grows.
- FID 1.24 is achieved by BitDance-H-1x with 1B parameters, outperforming prior autoregressive baselines under raster-scan decoding.
4.3 Text-to-image Generation
BitDance extends its binary-token autoregressive framework to text-to-image generation through multimodal training and parallel decoding. It reports strong prompt following and competitive performance with faster inference, including for 1024×1024 images.
- Resolution tokens specify visual-token grid dimensions, allowing the number of decoded tokens to vary with image resolution and aspect ratio.
- The text-to-image model uses a three-stage pre-training, continued-training, and supervised-fine-tuning pipeline, followed by optional distillation.
- 64-token parallel prediction is obtained from 16-token prediction through distillation using only a few training steps.
- The model is evaluated on prompt following, text rendering, and reasoning across GenEval, DPG-Bench, TIIF, and OneIG benchmarks.
- BitDance achieves state-of-the-art performance among autoregressive text-to-image models and is comparable to leading proprietary and diffusion models.
- GenEval 0.86 and DPG-Bench 88.28 place BitDance among the top-performing methods despite training on fewer than 450M image-text pairs.
- BitDance narrows the performance gap with commercial models on OneIG-EN and OneIG-ZH, particularly in text fidelity and alignment.
- The distilled model preserves excellent generation quality while achieving faster inference for 1024×1024 image generation.
4.4 Ablation Study
Ablations show that binary tokenization and diffusion-based sampling support BitDance’s generation quality and efficiency. Next-patch design choices improve performance, and few diffusion steps suffice for rapid sampling.
- Continuous VAEs perform notably worse than the binary tokenizer for autoregressive generation.The study attributes this to error accumulation from unconstrained continuous tokens.
- Token classification suffers out-of-memory failures, while bitwise classification performs poorly because it assumes independent bits.
- Patch-wise raster scanning and block-causal intra-patch visibility both improve next-patch diffusion performance.
- 10 diffusion sampling steps attain good results, indicating rapid sampling with binary tokens.
- Binary diffusion predictions become more distinct across timesteps and converge toward binary values of -1 and 1 as noise decreases.
- The binary diffusion head directly predicts clean binary latents through the x-prediction formulation.
5 Conclusion
BitDance is a scalable autoregressive model that generates high-quality images by efficiently predicting binary visual tokens. Its binary tokenizer, diffusion-based sampling, and parallel prediction are validated across class-conditional and text-to-image benchmarks.
- BitDance generates high-quality images through efficient prediction of binary visual tokens.
- The tokenizer expands visual vocabulary to 2^256, achieving reconstruction fidelity comparable to continuous VAEs.
- Binary diffusion and next-patch diffusion model joint distributions across multiple binary tokens for efficient, precise parallel prediction.
- Extensive evaluations on class-conditional and text-to-image benchmarks validate BitDance's effectiveness and efficiency.
- The authors propose scaling data and model size to explore BitDance in a wider range of multimodal tasks.