Source-linked AI summary
Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, Xiaobing Liu
TL;DR
High-resolution autoregressive image synthesis is limited by discrete-token quantization and visual train-test discrepancy. Infinity introduces bitwise token prediction with an infinite-vocabulary classifier and self-correction, and reports record autoregressive performance while surpassing leading diffusion models on several benchmarks. Its results support scaling tokenizer and transformer capacity for higher-quality text-to-image generation.
Problem
Autoregressive visual models face quantization errors in limited-vocabulary discrete tokenizers and train-test discrepancies that impair high-resolution image generation.
Method
Infinity uses bitwise visual tokens, an infinite-vocabulary classifier, bitwise self-correction, and jointly scaled tokenizer and transformer capacities.
Results
Infinity surpasses leading diffusion models on several text-to-image benchmarks, including improving SD3's GenEval from 0.62 to 0.73 and ImageReward from 0.87 to 0.96.
Takeaways & Limitations
Infinity demonstrates the potential of scaling visual tokenizers and transformers for near-continuous representation quality and high-quality text-to-image generation.
Abstract
from arXiv · showhide
We present Infinity, a Bitwise Visual AutoRegressive Modeling capable of generating high-resolution, photorealistic images following language instruction. Infinity redefines visual autoregressive model under a bitwise token prediction framework with an infinite-vocabulary tokenizer & classifier and bitwise self-correction mechanism, remarkably improving the generation capacity and details. By theoretically scaling the tokenizer vocabulary size to infinity and concurrently scaling the transformer size, our method significantly unleashes powerful scaling capabilities compared to vanilla VAR. Infinity sets a new record for autoregressive text-to-image models, outperforming top-tier diffusion models like SD3-Medium and SDXL. Notably, Infinity surpasses SD3-Medium by improving the GenEval benchmark score from 0.62 to 0.73 and the ImageReward benchmark score from 0.87 to 0.96, achieving a win rate of 66%. Without extra optimization, Infinity generates a high-quality 1024x1024 image in 0.8 seconds, making it 2.6x faster than SD3-Medium and establishing it as the fastest text-to-image model. Models and codes will be released to promote further exploration of Infinity for visual generation and unified tokenizer modeling.
1 Introduction
Infinity addresses high-resolution autoregressive image synthesis challenges by replacing index-wise visual tokens with bitwise modeling and scaling tokenizer and transformer capacity. It reports stronger text-to-image quality, prompt adherence, and inference speed than prior autoregressive and leading diffusion models.
- Autoregressive models struggle with high-resolution synthesis because discrete tokens reconstruct less accurately than continuous tokens.
- Limited index-wise tokenizer vocabularies introduce quantization errors that hinder fine-grained detail reconstruction in high-resolution images.
- Infinity replaces index-wise tokens with bitwise tokens through a bitwise visual tokenizer, infinite-vocabulary classifier, and bitwise self-correction.
- Infinity scales tokenizer vocabulary and transformer size, improving the content and details of generated images as both capacities increase.
- 0.62 to 0.73: Infinity improves SD3's GenEval score, while ImageReward rises from 0.87 to 0.96 and human evaluation reaches a 66% win rate.
- Infinity achieves exceptionally strong prompt adherence, superior image quality, and the fastest inference speed among the described autoregressive models.
2 Related Work
Related work contrasts diffusion's continuous denoising with autoregressive discrete-token generation and motivates scaling visual models. Infinity's framework is presented against these established approaches and the broader scaling-law literature.
- Autoregressive models convert images into discrete tokens and generate them through causal next-token or next-scale prediction.
- 2.2 Diffusion Models.: Diffusion models generate images by learning to invert data toward random noise through continuous denoising.
- Infinity's framework uses bitwise multi-scale visual tokens, an Infinite-Vocabulary Classifier, and Bitwise Self-Correction for next-scale prediction conditioned on text.
- Scaling laws in autoregressive language models relate model size, dataset size, and compute to test-set cross-entropy, inspiring scaling research in visual generation.
3 Infinity Architecture
Infinity combines multi-scale residual visual tokenization, bitwise prediction, and self-correction to address the computational, optimization, and train-test challenges of high-resolution autoregressive image generation.
- Visual AutoRegressive Modeling: Infinity encodes images into progressively higher-resolution multi-scale residual maps and reconstructs features from their upsampled cumulative sum.The transformer predicts residuals at each scale conditioned on previous predictions and text input.
- Visual AutoRegressive Modeling: Text embeddings guide residual prediction, while the transformer uses previous-scale features as inputs and predicts each later-scale residual in parallel.Flan-T5 text embeddings provide the conditioning signal through cross attention.
- Bitwise Visual Tokenizer and Classifier: A vocabulary of size Vd = 2^d makes conventional index-wise classifiers computationally prohibitive, including an 8.8-trillion-parameter example for Vd = 2^32 and h = 2048.Large vocabularies also create optimization difficulty because slight perturbations near zero can change discrete labels substantially.
- Bitwise Visual Tokenizer and Classifier: The bitwise tokenizer replaces index-wise quantization with binary codes, enabling extremely large vocabularies while reducing memory and computational demands.The method considers LFQ and BSQ; directly computing LFQ entropy over a codebook has O(2^d) complexity.
- Bitwise Self-Correction: Bitwise Self-Correction randomly flips residual bits, recomputes transformer inputs, and re-quantizes targets so training exposes the model to prediction errors.The procedure revises inputs and labels without adding computational cost or disrupting parallel training.
4 Experiment
Experiments evaluate Infinity’s image quality, prompt adherence, text rendering, benchmark performance, human preference, speed, scaling behavior, and ablations. Results show strong generation quality, scaling effects, and benefits from large vocabularies and bitwise self-correction.
- Qualitative Results: Infinity generates high-fidelity images across varied categories and consistently follows both short and extremely long user prompts.The reported improvements are attributed to bitwise token prediction and scaling autoregressive modeling.
- Qualitative Results: Infinity accurately renders requested text across diverse backgrounds and subjects, including specified fonts, styles, and colors.
- Benchmark: 0.73 is Infinity’s GenEval overall score, while its DPG overall score is 83.46 and its best relation score is 90.76 among open-source text-to-image models.Its GenEval position reasoning score is 0.49.
- Human Preference Evaluation: Infinity receives higher human preference for overall quality, prompt following, and visual aesthetics, and achieves the highest ImageReward and HPSv2.1 scores.The evaluation compares generated images from Infinity with other open-source text-to-image models.
- Ablation Studies: Increasing vocabulary size consistently improves tokenizer reconstruction, reaching rFID 0.61 on ImageNet 256×256 at Vd = 2^32.The discrete tokenizer surpasses the continuous VAE of Stable Diffusion in this setting.
- Scaling Effects: Scaling transformer size and training compute improves validation loss, evaluation metrics, and generated-image semantics and visual quality.GenEval, ImageReward, and HPSv2 correlate nearly linearly with validation loss, with Pearson coefficients of -0.983, -0.981, and -0.979.
- Ablation Studies: Bitwise self-correction substantially improves results over teacher forcing, whereas random bit flipping alone does not; a 30% flipping strength performs best.The mechanism imitates prediction errors and re-quantizes residuals to mitigate train-test discrepancy.
5 Conclusion
Infinity is introduced as a bitwise visual autoregressive model for text-to-image generation. The paper reports that it improves discrete generative models’ scaling and visual-detail capabilities and matches or surpasses leading diffusion models.
- Infinity combines bitwise token modeling with an infinite-vocabulary classifier and self-correction to improve scaling and visual detail representation.
- Extensive qualitative and quantitative results show Infinity matching or surpassing leading diffusion models for text-to-image generation.
- The authors expect Infinity to promote autoregressive visual modeling and faster, more realistic generation models.
A Predefined Scale Schedules
Infinity uses predefined scale schedules tailored to aspect ratios for coarse-to-fine next-scale prediction. These schedules support both standard and unusual image shapes while keeping training sequences efficient.
- Each aspect ratio has a predefined scale schedule containing height-width tuples for successive scales.
- The schedules approximately preserve the target aspect ratio, especially at later scales, while keeping same-scale areas roughly equal across aspect ratios.
- Bucketed schedules keep sequence lengths roughly consistent across aspect ratios, improving training efficiency.
- Infinity can generate images at common ratios such as 1:1, 16:9, and 4:3, as well as special ratios such as 1:3 and 3:1.
- K=13 scales are used to generate a 1024 × 1024 image or an image with another aspect ratio.
B Human Preference Evaluation
The human-preference study compares Infinity with four open-source text-to-image models using diverse prompts and blinded side-by-side judgments. Infinity is preferred across quality, prompt following, and visual aesthetics.
- Prompts: The evaluation uses 360 prompts spanning nine categories, including humans, animals, products, landscapes, architecture, plants, and text rendering.
- Prompts: Prompt challenges include complexity, quantity, positioning and perspective, style, detail, semantic understanding, color, and imagination.
- Compared Models: Infinity is compared with PixArt-Sigma, SD3-Medium, SDXL, and HART using official inference code without cherry-picking.
- Human Evaluation: Volunteers anonymously choose between paired images for overall quality, prompt following, and visual aesthetics, with each comparison evaluated by at least two volunteers.
- Results: Infinity achieves 90.0%, 83.9%, and 93.2% win rates against HART for overall quality, prompt following, and visual aesthetics, respectively.
- Results: Against diffusion models, Infinity achieves win rates of 76.0% versus PixArt-Sigma, 79.0% versus SDXL, and 66.0% versus SD3-Medium.Infinity also reaches a 71.1% win rate against SD3-Medium for visual aesthetics.
C More Qualitative Results
Infinity is qualitatively compared with four open-source models across text-to-image examples, showing stronger prompt adherence, human anatomy, and artistic-style rendering than several baselines.
- Infinity-2B is compared with Flux Schnell, SD3-Medium, PixArt Sigma, and HART in qualitative text-to-image examples.
- Infinity generates comparable or better images than diffusion models including Flux Schnell, SD3-Medium, and PixArt Sigma.
- Infinity follows text prompts better than SD3-Medium, HART, and PixArt Sigma in the first two examples.
- Infinity performs better on human hands and legs in the third and fourth examples.
- Infinity and PixArt Sigma successfully render an oil-painting style in the final example, while the other three models fail.