PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis

Patrick Esser, Robin Rombach, Andreas Blattmann, Björn Ommer

arXiv:2108.08827v1cs.CV

TL;DR

Autoregressive image models lack bidirectional and global context because they synthesize patches in a fixed sequential order. ImageBART reverses a multinomial diffusion hierarchy with autoregressive Markov-chain transitions over compressed discrete representations, improving image modification while retaining high-fidelity synthesis and controllable editing. Its reverse modeling assumes sufficiently expressive components, while conventional discrete diffusion would otherwise require roughly 1000 steps.

  • Problem

    Autoregressive image models use unidirectional context and a single-scale sequential factorization, limiting global contextual modeling and image modification.

  • Method

    ImageBART learns a compressed discrete representation and reverses a fixed multinomial diffusion hierarchy with autoregressive Markov-chain transitions that incorporate coarser global context.

  • Results

    ImageBART improves image modification capabilities while providing high-fidelity image generation and supporting global and local manipulation.

  • Takeaways & Limitations

    The hierarchical construction enables controlled image editing and conditional synthesis using autoregressive models in a compressed latent space.

  • Takeaways & Limitations

    Recovering the true data distribution requires reverse models expressive enough to represent the true reverse processes, while conventional discrete diffusion would require roughly 1000 steps.

Abstract

from arXiv · show

Autoregressive models and their sequential factorization of the data likelihood have recently demonstrated great potential for image representation and synthesis. Nevertheless, they incorporate image context in a linear 1D order by attending only to previously synthesized image patches above or to the left. Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete. It also processes the entire image on a single scale, thus ignoring more global contextual information up to the gist of the entire scene. As a remedy we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process: Whereas a multistage diffusion process successively removes information to coarsen an image, we train a (short) Markov chain to invert this process. In each stage, the resulting autoregressive ImageBART model progressively incorporates context from previous stages in a coarse-to-fine manner. Experiments show greatly improved image modification capabilities over autoregressive models while also providing high-fidelity image generation, both of which are enabled through efficient training in a compressed latent space. Specifically, our approach can take unrestricted, user-provided masks into account to perform local image editing. Thus, in contrast to pure autoregressive models, it can solve free-form image inpainting and, in the case of conditional models, local, text-guided image modification without requiring mask-specific training.

1 Introduction

ImageBART addresses the unidirectional context and limited global awareness of autoregressive image models with a coarse-to-fine hierarchy built from multinomial diffusion. The resulting model targets high-fidelity synthesis while improving image modification and conditional editing capabilities.

  • Autoregressive image models impose a top-left-to-bottom-right ordering that attends only to left or above context.
  • This ordering is perceptually unnatural, limits completions when only the lower half is available, and creates a training–inference gap through sequential sampling.
  • ImageBART uses a coarse-to-fine hierarchy in which compressed representations provide global context to autoregressive processes at finer stages.
  • The method inverts a fixed multinomial diffusion process with a Markov chain whose transitions are learned autoregressively.
  • The approach learns in a compact discrete representation space to support high-fidelity synthesis, conditional tasks, and locally controlled image editing.

2 Related Work

Prior work combines autoregressive modeling with learned image representations to improve synthesis, while latent-variable and diffusion models offer alternative ways to capture global or multiscale structure. ImageBART builds on these directions by applying multinomial diffusion to compressed discrete representations and using autoregressive reverse processes.

  • Latent-variable models use unobserved representations that can support sampling, interpolation, modification, and multiscale structure.
  • Autoregressive models are powerful density estimators but can produce samples lacking global consistency because of locality and sequential modeling biases.
  • Learning compact discrete image representations enables transformer-based autoregressive modeling and coherent high-resolution synthesis.
  • ImageBART combines these ideas through a compressed representation and a hierarchical autoregressive Markov-chain construction.
  • Pixel-space diffusion models can spend capacity on subtle details and require many reverse steps, hindering high-resolution applications.
  • Applying multinomial diffusion to compressed discrete representations, with autoregressively factorized reverse models, reduces diffusion steps and improves sampling speed.

3 Method

The method learns a compact discrete image representation, then constructs a hierarchy by progressively removing information with a fixed multinomial diffusion process. Autoregressive encoder-decoder transitions reverse this hierarchy while using coarser states as global context.

  • ImageBART models a sequence of distributions in which each finer representation is generated conditionally on a coarser representation.
  • A forward Markov chain progressively removes information, while a learned reverse Markov chain reconstructs the data distribution from the coarsest state.
  • The first hierarchy level uses a discrete autoencoder objective to learn a compressed representation without forcing later stages to model redundant image details.
  • Images are vector-quantized into discrete codebook indices and unrolled into sequences for autoregressive learning before CNN decoding.
  • Multinomial diffusion randomly replaces a fixed fraction of components, producing increasingly abstract representations controlled by β_t.
  • Autoregressive encoder-decoder transitions use cross-attention to obtain global context from the preceding hierarchy level, avoiding the need for roughly 1000 shared-weight reverse steps.

4 Experiments

Experiments evaluate ImageBART for unconditional and conditional synthesis, high-resolution generation, local editing, and sampling efficiency. Across these settings, the model produces coherent images, improves conditional metrics over TT, supports arbitrary-mask editing, and benefits from additional diffusion scales.

  • High-Fidelity Image Synthesis: ImageBART generates realistic and coherent 256 × 256 samples across FFHQ, LSUN, and ImageNet settings with a single reverse-chain pass.The evaluation covers unconditional generation on FFHQ, LSUN-Cats, LSUN-Churches, and LSUN-Bedrooms, plus class-conditional ImageNet synthesis.
  • Conditional Markov Chains for Controlled Image Synthesis: ImageBART improves FID, Inception Score, and CLIP text-image cosine similarity over TT for text-conditional synthesis.The model responds to semantic changes in conditioning sentences, such as weather changes in mountain imagery.
  • Resolutions Beyond 256×256 Pixels: ImageBART renders consistent megapixel images by applying its models patch-wise with sliding attention and progressively incorporating global context.The reported example is a 300 × 1800 image generated while interpolating between two text prompts.
  • Beyond Conditional Models: Local Editing with Autoregressive Models: Arbitrary masked reverse processes enable globally consistent upper-half completion and localized editing with free, hand-drawn masks.The approach does not require task-specific mask training and can combine local editing with class or text prompts.
  • Ablations: Modification performance increases monotonically with the number of diffusion scales, while unconditional-generation FID appears to plateau beyond T = 5.Both synthesis and modification are evaluated using FID with a single reverse Markov-chain pass.
  • Ablations: ImageBART is faster than pixel-based generative models, while its runtime increase relative to TT remains moderate through reduced decoder depth.Further decoder-layer reductions can increase runtime speed while maintaining satisfactory synthesis quality.

5 Conclusion

ImageBART introduces hierarchical bidirectional context into autoregressive transformer models for high-fidelity, controllable image synthesis. The approach supports global and local manipulation, conditional synthesis, and progressively compressed image representations.

  • ImageBART inverts a multinomial diffusion process with a Markov chain that incorporates context progressively from coarse to fine.
  • The hierarchy provides global context from previous levels, reducing the unnatural unidirectional ordering of pure autoregressive image models.
  • ImageBART enables global and local manipulation of input images, including conditional synthesis from varied representations.

A.1.1 Compression Models

The experiments use VQGAN-based image compression models, with dataset-specific training and effective codebook sizes. Compression-model hyperparameters are summarized in Table 4.

  • The image compression models are implemented as VQGANs using the official taming-transformers implementation.The models follow prior work and use the publicly available implementation as their starting point.
  • The LSUN compression model is fine-tuned from a publicly available model, whereas the FFHQ model is trained from scratch.
  • Unused codebook entries are removed by shrinking the codebook to its effective size before training the generative model.

A.1.2 Hierarchical Representations via Multinomial Diffusion

Multinomial diffusion creates a coarse-to-fine hierarchy while retaining a common spatial resolution across representations. The effective sequence length measures the valid entries remaining at each level.

  • All representations x_t for T > 1 have the same spatial resolution.
  • Each forward diffusion process gradually removes information, producing a coarse-to-fine hierarchy across levels.
  • At level x_t, the average number of valid entries is floor(ᾱ_t · N), defined as the effective sequence length.

A.1.3 Reverse Diffusion with Transformer Models

ImageBART reverses multinomial diffusion with learned sequence-to-sequence conditionals implemented using encoder-decoder models. The experiments specify process and model configurations across scales.

  • ImageBART is a learned Markov chain trained to reverse the multinomial diffusion process.
  • The reverse-diffusion conditionals are modeled efficiently with a sequence-to-sequence encoder-decoder architecture.
  • Tables 5 and 6 report the multinomial diffusion and Markov-chain hyperparameters used across experiments and scales.

A.1.4 Hardware

All listed models and the sampling-speed measurement used a single NVIDIA A100 GPU with 32-bit precision.

  • All models in Tables 4 and 6 were optimized on a single NVIDIA A100 GPU using 32-bit precision.
  • Sampling speed reported in Figure 8 was also measured on an NVIDIA A100.

A.2 Details on Conditional Experiments

ImageBART applies conditional synthesis across semantic maps, text, classes, and masks, using aligned conditioning and multiscale context to support high-resolution generation and local editing. The supplied experiments and discussion also identify resource, compression, and dataset-size constraints.

  • Semantically Guided Synthesis: Semantic landscape synthesis conditions ImageBART on discrete autoencoder representations of segmentation masks.Aligned image and mask embeddings are concatenated channel-wise to preserve the sequence length of the image representation.
  • Semantically Guided Synthesis: Sliding-window testing enables high-resolution landscape generation from models trained on 256 × 256 crops.
  • Masked Diffusion Processes for Local Editing: ImageBART uses global context from previous scales to produce consistent upper-half completions, whereas TT is limited to context from above.
  • Conditional Experiments: The model supports conditional synthesis and local editing across ImageNet, Conceptual Captions, semantic maps, and class-conditional exploration.Additional supplied figures show conditional inpainting, text-conditional samples, semantic synthesis, and class-conditional sample exploration.
  • Masked Diffusion Processes for Local Editing: Masked forward-backward diffusion preserves unmasked context while denoising only masked entries, enabling completions for arbitrary masks.The mask is downsampled to the representation size, with zero marking positions whose context is used and one marking positions where new content is generated.
  • Limitations and Societal Impacts: Strong image compression can cause reconstruction artifacts and may be unsuitable for very high-fidelity applications, while larger sequence lengths increase energy requirements.
Loading 2108.08827v1…