PaperlayerView exact revision on arXiv

Source-linked AI summary

Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning

Ting Chen, Ruixiang Zhang, Geoffrey Hinton

arXiv:2208.04202v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Discrete-data generation is difficult for autoregressive models because computation and memory can scale quadratically with data dimensions, while standard diffusion models assume continuous spaces. Bit Diffusion represents discrete data as real-valued analog bits and adds Self-Conditioning and Asymmetric Time Intervals. It achieves state-of-the-art discrete or categorical image generation, beats the best autoregressive model, and is competitive for MS-COCO image-conditional text generation.

  • Problem

    Autoregressive discrete-data models face computation and memory costs that can grow quadratically with sequence length or image size.

  • Method

    Bit Diffusion encodes discrete data into binary bits, models those bits as real-valued analog bits with continuous diffusion, and thresholds generated values for decoding.

  • Results

    The method achieves state-of-the-art discrete or categorical image generation and beats the best autoregressive model, while remaining competitive for MS-COCO image-conditional text generation.

  • Takeaways & Limitations

    The approach provides a simple, generic framework for applying continuous-state diffusion models to discrete data.

  • Takeaways & Limitations

    A reported implementation uses a model about 30M parameters smaller than Nichol and Dhariwal (2021), which may have a minor effect on performance.

Abstract

from arXiv · show

We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to a significant improvement in sample quality. Despite its simplicity, the proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For discrete image generation, we significantly improve previous state-of-the-art on both CIFAR-10 (which has 3K discrete 8-bit tokens) and ImageNet-64x64 (which has 12K discrete 8-bit tokens), outperforming the best autoregressive model in both sample quality (measured by FID) and efficiency. For image captioning on MS-COCO dataset, our approach achieves competitive results compared to autoregressive models.

1 INTRODUCTION

Bit Diffusion enables continuous-state diffusion models to generate discrete data by modeling binary representations as real-valued analog bits. The approach addresses scalability challenges in autoregressive models and achieves strong results in discrete image generation and image captioning.

  • Autoregressive models for discrete data can require computation and memory quadratic in sequence length or image size.
  • Diffusion models offer more parallel sampling and can model higher-dimensional data, but standard formulations assume continuous data and state spaces.
  • Analog bits cast binary representations of discrete data into real numbers for continuous-state diffusion modeling, then decode them by thresholding.
  • Self-Conditioning and Asymmetric Time Intervals are proposed as techniques that improve diffusion sample quality.
  • FID 6.93 on categorical CIFAR-10 surpasses the autoregressive model's 12.75 using one-third the model size and 100 rather than 3072 sequential inference steps.
  • On MS-COCO image captioning, Bit Diffusion achieves results competitive with a strong Transformer-based autoregressive captioner.

2 METHOD

The method represents discrete variables with real-valued analog bits and applies continuous diffusion training and sampling. It further improves generation by conditioning on previous estimates and manipulating time intervals during large reverse steps.

  • Preliminaries: Diffusion models learn reverse state transitions from noise to data by training a denoising network to estimate x0 or noise from xt.
  • Analog Bits: A discrete variable with alphabet size K is represented using n = ⌈log2 K⌉ binary bits, which are cast into R^n as analog bits.
  • Analog Bits: Generated analog bits are expected to form bimodal concentrations near binary values and can be decoded into discrete variables by thresholding.
  • Self-Conditioning: Self-Conditioning lets the denoising network receive its previously estimated sample as an additional input during iterative sampling.
  • Asymmetric Time Intervals: Asymmetric Time Intervals manipulate generation-time scheduling so large reverse steps use different time intervals for state transitions and time reduction.
  • Putting it together: The algorithms use a continuous time parameterization between 0 and 1 rather than fixed discrete time steps.

3 EXPERIMENTS

The experiments evaluate Bit Diffusion on discrete image generation and image captioning, using binary encodings, diffusion architectures, and comparisons with autoregressive and diffusion baselines. Results show strong discrete-image performance, benefits from Self-Conditioning and asymmetric time intervals, robust analog-bit concentration, and captioning performance comparable to autoregressive modeling.

  • Experimental tasks and datasets: Experiments cover discrete or categorical image generation and image-conditional text generation, using CIFAR-10, ImageNet 64×64, and MS-COCO captioning.FID is the main image-generation metric, computed between 50K generated samples and the training set.
  • Representations: Images use UINT8, GRAY CODE, or UINT8 (RAND) 8-bit sub-pixel encodings, while captions use 15 analog bits per token for sequences up to 960 bits.UINT8 and GRAY CODE retain weak ordinal structure, whereas UINT8 (RAND) removes correlation with intensity.
  • Discrete image generation: Bit Diffusion achieves better results than existing discrete diffusion and autoregressive models on CIFAR-10, while UINT8 and GRAY CODE achieve performance similar to continuous DDPM.The CIFAR-10 comparison reports state-of-the-art FIDs for discrete image generation.
  • Discrete image generation: Continuous-pixel diffusion has the best ImageNet FID, whereas UINT8 (RAND) has the worst, showing the difficulty introduced by removing sub-pixel intensity and order information.The comparison uses continuous-pixel DDPM because competing discrete ImageNet methods did not report FIDs.
  • Ablations: Self-Conditioning improves diffusion results across binary encodings and also benefits continuous diffusion, while predicting x0 is more effective than predicting ϵ for Bit Diffusion.The ablation compares settings that differ only in whether Self-Conditioning is used.
  • Ablations: With 100 ImageNet sampling steps, a non-zero time difference improves FID; as steps increase from 100 to 400, the optimal time difference shrinks to zero.For UINT8 (RAND), 400 steps can substantially worsen sample quality relative to 100 steps, and alternative strategies may improve FIDs at longer sampling times.
  • Analog-bit behavior: Generated analog bits concentrate around two modes despite lacking a hard bimodality constraint, making thresholding or quantization easy and robust.This distribution is visualized using 64 generated ImageNet images.
  • Image captioning: Bit Diffusion achieves similar image-captioning performance to the autoregressive model and generally needs about 10 inference steps for good results.Too few sampling steps can cause bit-level errors that produce substantially different words.

4 RELATED WORK

Related work contrasts autoregressive models, discrete diffusion extensions, flows, VAEs, and GANs with the paper’s continuous-state analog-bit approach. The central distinctions concern scalability, state-space flexibility, and the ease of quantizing bimodal analog bits.

  • Autoregressive models for discrete data: Autoregressive models dominate discrete-data generation but their computation and memory requirements typically grow quadratically with sequence length or image resolution.This scaling makes them challenging to apply to high-dimensional discrete data.
  • Diffusion models for discrete data: Existing diffusion models for discrete data reformulate diffusion in discrete data and state spaces, whereas continuous state spaces are described as more flexible and potentially more efficient.Standard state-of-the-art continuous diffusion models cannot directly generate discrete or categorical data.
  • Diffusion models for discrete data: Analog bits provide a fixed binary encoder whose bimodal real-valued outputs can be decoded by simple thresholding, unlike continuous embeddings that may have multiple modes per dimension.The comparison concerns the difficulty of quantization after generation.
  • Normalizing flows for discrete data: Discrete normalizing flows address the difficulty of applying invertible continuous transformations directly to categorical data by introducing invertible transformations for discrete variables.The supplied discussion frames this as a separate family of approaches to discrete generation.
  • Other generative models for discrete data: VAEs and GANs have been applied to discrete data but have not matched autoregressive models on discrete image or text generation in sample quality or related performance.The passage gives a broad comparison rather than a task-specific metric.
  • Other related work: Self-Conditioning shares similarities with GAN self-modulation and SUNDAE’s inference-step denoising mechanism.These approaches connect the technique to earlier methods that reuse or incorporate intermediate states during generation.

5 CONCLUSION

The paper encodes discrete data as real-valued analog bits for continuous diffusion models, adds Self-Conditioning and Asymmetric Time Intervals, and reports strong results across discrete image generation and image-conditional text generation.

  • 5 CONCLUSION: Analog bits encode discrete or categorical data as real numbers that continuous state diffusion models can generate, followed by decoding into discrete variables.The approach uses binary encodings and can also use one-hot encodings with arg max decoding.
  • 5 CONCLUSION: Self-Conditioning and Asymmetric Time Intervals are proposed as techniques that improve diffusion-model sample quality.Self-Conditioning conditions the model on previously generated samples during iterative sampling.
  • 5 CONCLUSION: The approach achieves state-of-the-art results in discrete or categorical image generation, beating the best autoregressive model.The paper also reports competitive results against autoregressive models for image-conditional text generation on MS-COCO.
  • 5 CONCLUSION: A limitation is that diffusion models still require a significant number of inference steps to generate good image samples.The paper expects future improvements for continuous diffusion models to transfer to discrete data using analog bits.
  • 5 CONCLUSION: One-hot encoding represents each discrete variable with K bits and decodes generated analog bits using arg max.Base-2 encoding requires only ⌈log2 K⌉ bits, making one-hot encoding less efficient for large vocabularies.
  • 5 CONCLUSION: The analog-bit approach can use alternative loss functions, including sigmoid and softmax cross entropy, in addition to the default ℓ2 loss.For one-hot analog bits, the paper describes softmax activation and softmax cross entropy loss.

B.4 PRELIMINARY EXPERIMENTS

The preliminary experiments report FIDs for Bit Diffusion models using different analog-bit types and loss functions on unconditional CIFAR-10, while noting that further tuning may improve results.

  • B.4 PRELIMINARY EXPERIMENTS: Table 6 reports FIDs for Bit Diffusion models with different analog-bit types and loss functions on unconditional CIFAR-10.The listed comparison concerns model variants rather than a single fixed analog-bit configuration.
  • B.4 PRELIMINARY EXPERIMENTS: The authors caution that some results may improve with additional hyperparameter tuning or network tweaks.These improvements are outside the focus of the work.

C ON BINARY ENCODING OF PIXELS: UINT8, GRAY CODE, UINT8 (RAND)

The appendix compares binary pixel encodings by relating subpixel-intensity differences to Hamming distances, finding that randomization removes the order information present in other encodings.

  • C ON BINARY ENCODING OF PIXELS: UINT8, GRAY CODE, UINT8 (RAND): UINT8 (RAND) is constructed by randomly permuting the 256 subpixel values before assigning the permuted integers’ binary bits.The described implementation uses numpy.random.seed(42) before shuffling.
  • C ON BINARY ENCODING OF PIXELS: UINT8, GRAY CODE, UINT8 (RAND): The comparison computes absolute subpixel-intensity differences and Hamming distances for every pair of integers in [0, 256).This procedure evaluates how binary encodings preserve relationships among pixel intensities.
  • C ON BINARY ENCODING OF PIXELS: UINT8, GRAY CODE, UINT8 (RAND): UINT8 and GRAY CODE show partial correlation between Hamming distance and subpixel intensity difference.Their different correlation patterns indicate that these encodings partially retain order information.
  • C ON BINARY ENCODING OF PIXELS: UINT8, GRAY CODE, UINT8 (RAND): UINT8 (RAND) shows no correlation between Hamming distance and subpixel intensity, so the original order information is fully removed.The appendix therefore treats UINT8 (RAND) as categorical data.
  • C ON BINARY ENCODING OF PIXELS: UINT8, GRAY CODE, UINT8 (RAND): Figure 9 illustrates how a continuous generative model can map a prior distribution into a binary data distribution.With a deterministic sampler, dividing the prior into two probability regions can represent any Bernoulli distribution.

E ON OTHER SAMPLERS FOR CONTINUOUS DIFFUSION MODELS

Because analog-bit models use continuous diffusion states, the paper explores DPM-Solver and finds fewer function evaluations can achieve comparable FIDs, while Self-Conditioning remains beneficial.

  • E ON OTHER SAMPLERS FOR CONTINUOUS DIFFUSION MODELS: DPM-Solver provides a boost to diffusion models based on analog bits, similar to its effect on continuous data.The authors identify this as a potential advantage over baselines using discrete states.
  • E ON OTHER SAMPLERS FOR CONTINUOUS DIFFUSION MODELS: 30 NFEs with DPM-Solver obtain comparable FIDs to 100 NFEs with DDIM for bit diffusion on discrete or categorical data.This result is reported for ImageNet-64x64 models under different binary encoding schemes.
  • E ON OTHER SAMPLERS FOR CONTINUOUS DIFFUSION MODELS: Self-Conditioning consistently improves DPM-Solver performance at a fixed number of function evaluations.The appendix reports this comparison for continuous RGB ImageNet-64x64 diffusion models.

F EXTRA RANDOM SAMPLES ON CIFAR-10 AND IMAGENET 64×64

The section presents additional CIFAR-10 and ImageNet 64×64 samples alongside sampling strategies built around Self-Conditioning. These strategies include momentum estimates, self-guidance, asymmetric time intervals, and binary decoding.

  • Additional random samples: Random samples are shown for unconditional CIFAR-10 and class-conditional ImageNet 64×64 models with continuous and discrete image generation.The figures report non-cherry-picked samples for both datasets and model types.
  • Self-Conditioning strategies: Momentum-based Self-Conditioning uses an exponential moving average of previous data estimates as the denoising network’s conditioning input.The default strategy is recovered when the momentum is set to zero.
  • Self-Conditioning strategies: Self-Guidance performs two denoising-network passes per sampling step and combines predictions from zero and current-step self-conditioning signals.This approach addresses the discrepancy between self-conditioning during training and sampling.
  • Binary decoding: Generated analog bits are converted into discrete data by thresholding predictions at zero and applying bit2int.This binary decoding step returns integer-valued discrete variables.
  • Sampling procedure: Asymmetric time intervals modify the next-state time using a time difference during iterative DDIM or DDPM sampling.The sampling procedures compute the current and next times before estimating the next latent state.

G.2 EXPERIMENTS

The experiments compare Self-Conditioning sampling strategies for conditional ImageNet 64×64 generation. Momentum helps mainly with larger sampling budgets, while self-guidance performs robustly with guidance weights between 3.0 and 5.0.

  • Experimental comparison: Table 9 reports the best FID scores across Self-Conditioning strategies, samplers, sampling steps, and asymmetric time differences.The comparison aggregates the strongest results for the evaluated sampling configurations.
  • Momentum Estimate: Momentum on the data estimate is helpful only when the number of sampling steps is larger.This finding comes from FIDs for conditional ImageNet 64×64 with UINT8 encoding.
  • Self-Guidance: A self-guidance weight between 3.0 and 5.0 is generally preferable and robust to sampler choice, sampling steps, and time difference.The result is reported for conditional ImageNet 64×64 with UINT8 encoding.

G.3 SAMPLES

The section presents generated ImageNet 64×64 samples for multiple Self-Conditioning strategies and sampling configurations. FID statistics are aggregated across sampling steps and asymmetric time differences for momentum and self-guidance variants.

  • Generated samples: Figures 14 and 15 compare generated categorical ImageNet 64×64 samples across Self-Conditioning strategies using 100 and 1000 DDIM steps.The compared samples use UINT8 (RAND) encoding and share the same initial x_T values across plots.
  • FID evaluations: Momentum-based FID statistics aggregate results over 100–1000 sampling steps and time differences from 0.0 to 0.8.The evaluated step counts are 100, 200, 400, 600, 800, and 1000.
  • FID evaluations: Self-guidance FID statistics aggregate results over 100–1000 sampling steps and time differences of 0.0 and 0.1.The evaluation uses conditional ImageNet 64×64 with UINT8 (RAND) encoding.
Loading 2208.04202v2…