PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoregressive Image Generation with Masked Bit Modeling

Qihang Yu, Qihao Liu, Ju He, Xinyang Zhang, Yang Liu, Liang-Chieh Chen, Xi Chen

arXiv:2602.09024v1cs.CV

TL;DR

The paper asks whether discrete visual generation is intrinsically worse than continuous generation and identifies latent bit allocation as a major factor in the gap. It introduces BAR, which generates token bits progressively with a masked bit modeling head to support large codebooks. BAR reaches gFID 0.99 on ImageNet-256 and establishes state-of-the-art performance across discrete and continuous methods while improving generation efficiency.

  • Problem

    Discrete tokenizers often use substantially higher compression ratios, while scaling their codebooks makes conventional discrete generators costly and difficult to optimize.

  • Method

    BAR equips an autoregressive transformer with a masked bit modeling head that iteratively generates the constituent bits of each discrete token.

  • Results

    gFID 0.99: BAR establishes a new state-of-the-art result on ImageNet-256 across discrete and continuous paradigms.

  • Takeaways & Limitations

    Under comparable bit budgets, discrete tokenizers can match or outperform continuous counterparts, and masked bit autoregressive modeling enables efficient generation with arbitrarily large vocabularies.

Abstract

from arXiv · show

This paper challenges the dominance of continuous pipelines in visual generation. We systematically investigate the performance gap between discrete and continuous methods. Contrary to the belief that discrete tokenizers are intrinsically inferior, we demonstrate that the disparity arises primarily from the total number of bits allocated in the latent space (i.e., the compression ratio). We show that scaling up the codebook size effectively bridges this gap, allowing discrete tokenizers to match or surpass their continuous counterparts. However, existing discrete generation methods struggle to capitalize on this insight, suffering from performance degradation or prohibitive training costs with scaled codebook. To address this, we propose masked Bit AutoRegressive modeling (BAR), a scalable framework that supports arbitrary codebook sizes. By equipping an autoregressive transformer with a masked bit modeling head, BAR predicts discrete tokens through progressively generating their constituent bits. BAR achieves a new state-of-the-art gFID of 0.99 on ImageNet-256, outperforming leading methods across both continuous and discrete paradigms, while significantly reducing sampling costs and converging faster than prior continuous approaches. Project page is available at https://bar-gen.github.io/

1. Introduction

The paper argues that discrete visual tokenizers are not intrinsically inferior to continuous ones: their observed gap largely reflects lower latent bit budgets. BAR addresses the resulting scalability challenge and achieves strong quality and efficiency results.

  • 1. Introduction: The performance gap between discrete and continuous pipelines is largely associated with latent-space compression rates rather than representation type.The paper compares both paradigms using the number of bits allocated to the latent space.
  • 1. Introduction: Allocating more bits per token, equivalent to scaling the codebook, allows discrete tokenizers to match or surpass continuous counterparts in reconstruction quality.
  • 1. Introduction: Larger codebooks improve reconstruction quality but make standard discrete generation memory-intensive, computationally complex, and difficult to optimize.The resulting vocabulary-scaling problem motivates a different prediction head.
  • 1. Introduction: BAR predicts discrete tokens by progressively generating their constituent bits instead of classifying over a massive vocabulary.This lightweight bit-generation mechanism is designed to accommodate unbounded vocabulary sizes.
  • 1. Introduction: gFID 0.99: BAR establishes a new state-of-the-art result on ImageNet-256 across discrete and continuous paradigms.A 415M-parameter variant reaches gFID 1.13 and is 3.68× faster than leading continuous approaches; the best variant reaches 0.99.

2. Related Work

Related work spans continuous VAE-and-diffusion pipelines and discrete tokenizer-and-generator approaches. BAR is positioned against methods that either predict codebook indices or use bit representations without fully resolving scalable generation.

  • 2. Related Work: Continuous visual generation typically combines a VAE for latent representation with a diffusion model for synthesis.VAEs downsample spatial dimensions while expanding channels to form compact representations suited to diffusion.
  • 2. Related Work: Discrete visual generation builds on VQGAN and studies learnable codebooks, improved quantization dynamics, and lookup-free quantizers.
  • 2. Related Work: MaskBit feeds bit tokens into its generator but still predicts codebook indices, limiting scalability with codebook size like standard discrete models.
  • 2. Related Work: Infinity generates images directly from bits and supports codebook size 2^64, but its approach differs from BAR's masked bit autoregressive formulation.The supplied passage introduces Infinity as a closely related large-codebook method.

3. Method

The method compares discrete and continuous visual tokenizers using bit budget, then scales discrete capacity and introduces BAR to make large-codebook generation tractable. BAR combines autoregressive context modeling with a masked bit prediction head, while experiments show improved reconstruction as bit allocation increases.

  • Tokenizer background: Visual tokenizers use an encoder, bottleneck, and decoder; discrete bottlenecks select finite codebook entries, while continuous bottlenecks use dimensionality reduction and regularization.Both paradigms produce latent representations for subsequent generation, but their bottleneck constraints differ.
  • Bit-budget comparison: Bit Budget measures latent information capacity and provides a common proxy for comparing discrete and continuous tokenizer compression.For continuous tokenizers, the formulation uses latent channel dimension and 16 bits per channel under mixed-precision training; effective information can be lower than nominal capacity.
  • Bit-budget comparison: Discrete tokenizers generally reconstruct worse while using fewer bits, and increasing latent bit allocation progressively narrows their gap with continuous tokenizers.The comparison attributes much of the observed performance difference to compression ratio rather than an intrinsic representation disadvantage.
  • Scaling discrete tokenizers: BAR-FSQ scales discrete codebooks with FSQ, varying latent channel dimensions from 10 to 256 and codebook sizes from 2^10 to 2^256.The experiments fix 256 latent tokens and use 1 bit per channel while varying channel dimension.
  • Scaling discrete tokenizers: At 65536 bits, BAR-FSQ achieves rFID 0.33 versus SD-VAE rFID 0.62, while at 16384 bits it achieves rFID 0.50.Reconstruction quality improves consistently with codebook size, reaching competitive or superior fidelity beyond a certain bit budget.
  • BAR framework: BAR replaces the vocabulary-sized linear head with a lightweight masked bit modeling head that progressively unmasks token bits, supporting arbitrarily large vocabularies.The autoregressive transformer models global structure, while the MBM head treats token prediction as conditional bit generation rather than massive classification.

4. Experimental Results

Experiments show that masked bit modeling scales discrete generation to large codebooks while preserving strong quality and enabling favorable accuracy–cost trade-offs. BAR achieves state-of-the-art ImageNet-256 results and substantially faster sampling than prior methods.

  • Different Prediction Heads: MBM scales to arbitrary codebook sizes and achieves gFID 1.37 with codebook size 2^32.Linear heads become infeasible at large vocabularies, while conventional bit heads show substantial quality degradation.
  • Sampling Strategy: Uniform sampling performs strongly across strategies, while logit-normal sampling provides a slight advantage, especially for larger codebook sizes.BAR remains robust across the evaluated sampling strategies.
  • Prediction Head Size: Increasing prediction-head capacity improves generation quality without CFG, with larger benefits at larger codebook sizes.The gains become less pronounced when classifier-free guidance is applied.
  • Sampling Strategy: Three sampling steps substantially improve quality over two, whereas additional steps provide only marginal gains; back-loading helps with CFG but slightly hurts without it.The reported ablation compares both sampling-step counts and bit-unmasking schedules.
  • Efficient Generation with Token-Shuffling: BAR-B/2 reduces tokens fourfold and raises throughput from 24.9 to 150.3 images per second, with modest gFID degradation.More aggressive BAR-B/4 downsampling reaches 445.5 images per second.
  • ImageNet 256×256: BAR-B reaches gFID 1.13 with 415M parameters, outperforming RAR, VAR, LlamaGen, and several continuous baselines.Its 415M parameters are one quarter of RAR’s 1.5B, while its gFID is 1.13 versus RAR’s 1.48.
  • ImageNet 256×256: BAR-L achieves a state-of-the-art gFID of 0.99 and establishes the best reported result without guidance at gFID 1.42.BAR-L outperforms prior discrete and continuous methods in both settings.
  • Sampling Speed: BAR-B/2 achieves gFID 1.35 with 150.52 images per second, while BAR-B/4 samples 2.94× faster than MeanFlow at comparable quality.BAR-B/2 is also 30.59× faster than PAR and 18.64× faster than VAR.

5. Conclusion

The paper finds that discrete tokenizers can match or outperform continuous tokenizers under comparable bit budgets, and introduces masked bit autoregressive modeling for efficient large-vocabulary generation. The work reports broader computational benefits and potential accessibility gains for visual generation.

  • Comparable bit budgets allow discrete tokenizers to match or outperform continuous counterparts.
  • BAR models discrete tokens by generating token bit representations, enabling efficient generation with arbitrarily large vocabularies.
  • BAR substantially outperforms existing discrete methods and strong continuous baselines.
  • The approach is described as enabling faster sampling and more efficient training, potentially improving computational accessibility and reducing environmental impact.

A. Appendix

The appendix provides supplementary implementation details, additional ImageNet-512 experiments, and more visualizations of BAR models.

  • Section B provides hyper-parameters for the final BAR-FSQ and BAR models.
  • Section C reports additional BAR results on the ImageNet-512 benchmark.
  • Section D provides additional visualization samples of BAR models.

B. Hyper-parameters for Final BAR Models

The final BAR model configurations and associated training and sampling hyper-parameters are documented in the appendix tables.

  • Table 5 details the architecture configurations of BAR models.
  • The BAR model configuration is detailed in Table 5.
  • Tables 6 and 7 list training and sampling hyper-parameters for BAR-FSQ and BAR models.

C. More Experimental Results

Additional experiments report BAR results on ImageNet-512 and describe the evaluation setup and computational constraints for those results.

  • BAR demonstrates clear advantages over other methods on ImageNet-512.
  • ImageNet-512 results use classifier-free guidance, with BAR using a simple linear guidance schedule.
  • BAR does not require the external-model auto-guidance used by other listed state-of-the-art methods.
  • The ImageNet-512 model was trained for 200 epochs because of computational constraints.

D. Visualization on Generated Samples

Figures 7–18 present visualization results from BAR models across diverse ImageNet classes. The captions describe the generated samples as high-fidelity and diverse.

  • D. Visualization on Generated Samples: BAR generates high-fidelity image samples with great diversity.This description is repeated across the visualization figure captions.
  • D. Visualization on Generated Samples: The visualized classes include goldfish, loggerhead turtle, lorikeet, jellyfish, and golden retriever.These examples span animal categories represented in the generated samples.
  • D. Visualization on Generated Samples: Additional visualized classes include Siberian husky, balloon, fountain, ice cream, cheeseburger, bubble, and volcano.The figures also show generated samples for object, food, and natural-scene categories.
Loading 2602.09024v1…