Source-linked AI summary
MaskGIT: Masked Generative Image Transformer
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman
TL;DR
Existing generative transformers decode raster-ordered image tokens sequentially, limiting efficiency and treating images as flat sequences. MaskGIT uses bidirectional masked-token prediction with iterative parallel refinement, achieving up to 64x faster ImageNet decoding and extending to image editing tasks. The paper reports competitive image-editing performance and higher-quality ImageNet samples, while noting limitations in attention size and complex structures.
Problem
Generative transformers flatten images into raster-ordered token sequences and decode them sequentially, a strategy the paper describes as neither optimal nor efficient for images.
Method
MaskGIT trains a bidirectional transformer on randomly masked visual tokens and generates images through iterative parallel prediction, confidence-based retention, and remasking.
Results
On ImageNet, MaskGIT generates higher-quality samples than VQGAN and accelerates autoregressive decoding by up to 64x.
Takeaways & Limitations
MaskGIT extends to image manipulation, inpainting, and extrapolation, obtaining comparable performance to dedicated models on image-editing tasks.
Takeaways & Limitations
Limited attention size can cause semantic or color shifts, boundary-object changes, and artifacts or oversmoothing on complex structures.
Abstract
from arXiv · showhide
Generative transformers have experienced rapid popularity growth in the computer vision community in synthesizing high-fidelity and high-resolution images. The best generative transformer models so far, however, still treat an image naively as a sequence of tokens, and decode an image sequentially following the raster scan ordering (i.e. line-by-line). We find this strategy neither optimal nor efficient. This paper proposes a novel image synthesis paradigm using a bidirectional transformer decoder, which we term MaskGIT. During training, MaskGIT learns to predict randomly masked tokens by attending to tokens in all directions. At inference time, the model begins with generating all tokens of an image simultaneously, and then refines the image iteratively conditioned on the previous generation. Our experiments demonstrate that MaskGIT significantly outperforms the state-of-the-art transformer model on the ImageNet dataset, and accelerates autoregressive decoding by up to 64x. Besides, we illustrate that MaskGIT can be easily extended to various image editing tasks, such as inpainting, extrapolation, and image manipulation.
1. Introduction
Existing generative transformers flatten images into raster-ordered token sequences and decode them sequentially, a strategy described as inefficient for images. MaskGIT instead uses bidirectional iterative parallel decoding, achieving faster and higher-quality ImageNet synthesis while extending to image editing.
- Motivation: Generative transformers commonly represent images as 1D raster-scan token sequences and generate them autoregressively.This inherited sequential formulation differs from the non-sequential structure described for images.
- Approach: MaskGIT trains a bidirectional transformer to predict masked tokens and iteratively retains confident parallel predictions while remasking the rest.The mask ratio decreases until all tokens are generated.
- Efficiency: 8 iterations replace 256 sequential rounds in the illustrated MaskGIT decoding comparison.MaskGIT predicts scattered tokens in parallel, with the number of predicted tokens increasing over iterations.
- Results: Up to 64x faster decoding and higher-quality samples are reported for MaskGIT than for VQGAN on ImageNet class-conditional generation.The comparison covers 256x256 and 512x512 resolutions.
- Image editing: MaskGIT is readily extended to class-conditional editing, inpainting, and extrapolation, with comparable performance to dedicated models on the latter tasks.The model can regenerate masked content while preserving surrounding context.
2. Related Work
Prior image synthesis systems commonly use a two-stage latent-token pipeline, while masked modeling has mainly supported representation learning rather than image generation. MaskGIT applies bidirectional masked modeling to generation and presents evidence on ImageNet.
- Image Synthesis: VQVAE-style image synthesis compresses images into discrete latent tokens before modeling their latent priors and decoding them into pixels.The first stage uses an encoder, codebook, and decoder.
- Image Synthesis: Generative transformer systems generally retain autoregressive sequential token prediction after image tokenization.Existing work focuses heavily on reducing information loss in tokenization while sharing the second-stage sequence model.
- Masked Modeling: BERT-style masked modeling predicts masked elements using bidirectional context, and vision methods extended this setup to quantized image tokens for representation learning.The cited masked modeling uses context from both directions.
- MaskGIT: MaskGIT's pipeline combines a visual tokenizer with a bidirectional transformer performing Masked Visual Token Modeling.The transformer learns to predict randomly masked visual tokens.
- Masked Modeling: Few works had successfully applied masked modeling to image generation because bidirectional attention complicates autoregressive decoding.The paper identifies its ImageNet results as first evidence of masked modeling efficacy for image generation on that benchmark.
3. Method
MaskGIT improves the latent-token generation stage with bidirectional masked-token training and iterative parallel decoding. Its masking schedule controls how many tokens remain masked during training and inference, with cosine scheduling reported as the best tested option.
- Method: MaskGIT keeps the VQGAN tokenizer setup and replaces the second-stage autoregressive model with a bidirectional transformer trained by MVTM.Potential improvements to tokenization are left for future work.
- 3.1. MVTM in Training: MVTM randomly replaces selected visual tokens with [MASK] and trains the transformer to predict the masked tokens from bidirectional context.The objective is the cross-entropy between ground-truth tokens and predicted probabilities.
- 3.2. Iterative Decoding: At inference, all masked positions are predicted simultaneously, sampled tokens receive confidence scores, and only the most confident predictions are retained.Unmasked positions receive confidence 1.0, while remaining tokens are remasked for later refinement.
- 3.2. Iterative Decoding: The decoding process starts from an all-masked canvas and progressively reduces the mask ratio until all tokens are generated within T iterations.The procedure is iterative because a single-pass prediction is challenging due to mismatch with the training task.
- 3.3. Masking Design: The mask schedule γ determines the masked-token count as n = round(γ(t/T)N), where N is input length and T is total iterations.The schedule is used in both training and inference to represent decoding progress.
- 3.3. Masking Design: Cosine scheduling achieves the best FID among the tested masking functions.The paper contrasts linear, concave, and convex schedules; cosine belongs to the concave family.
4. Experiments
MaskGIT is evaluated for image-generation quality, speed, editing flexibility, and mask-scheduling choices. Across ImageNet and image-editing tasks, it improves quality and diversity over key baselines while substantially reducing decoding runtime.
- Class-conditional Image Synthesis: MaskGIT significantly outperforms VQGAN in ImageNet class-conditional synthesis quality at 256ˆ256 and 512ˆ512 resolutions.The evaluation compares quality using FID and IS, including a VQGAN baseline trained with the same tokenizer and hyperparameters.
- Class-conditional Image Synthesis: 30-64x runtime acceleration over VQGAN becomes more pronounced as image resolution and token length increase.Speed is assessed through decoding runtime and the number of neural-network forward passes required to generate a sample.
- Class-conditional Image Synthesis: MaskGIT establishes state-of-the-art CAS on ImageNet at both evaluated resolutions and improves diversity through better recall than VQGAN.Its samples also achieve better coverage than BigGAN, while slightly boosting VQGAN’s precision.
- Image Editing Applications: Without architecture or task-specific training, MaskGIT supports class-conditional editing, inpainting, and outpainting through constraints on the initial decoding mask.For class-conditional editing, the model regenerates a boxed region for a target class while preserving surrounding context.
- Image Editing Applications: On Places2 inpainting with a central 50% ˆ 50% mask, MaskGIT beats DeepFillv2 and HiFill in FID and IS while approaching CoModGAN.The outpainting adaptation uses the same problem and model setup as inpainting, despite having fewer surrounding-pixel constraints.
- Ablation Studies: Cosine mask scheduling achieves the best FID and becomes MaskGIT’s default, with a reported sweet spot at 8 to 12 decoding iterations.More iterations are not necessarily better; performance peaks and then worsens, with the peak depending on the scheduling function.
5. Conclusion
MaskGIT introduces bidirectional transformer decoding with iterative generation in a constant number of iterations. It outperforms state-of-the-art transformer models on conditional image generation and extends to image manipulation tasks.
- MaskGIT uses a bidirectional transformer decoder and Masked Visual Token Modeling for iterative image generation.
- MaskGIT generates samples through an iterative decoding process within a constant number of iterations.
- MaskGIT significantly outperforms the state-of-the-art transformer model on conditional image generation.
- MaskGIT is readily extendable to various image manipulation tasks.
- The authors identify applying MaskGIT to other synthesis tasks as a promising direction because it achieves competitive performance with state-of-the-art GANs.
A. Discussion on Image Reconstruction
MaskGIT reconstructs images from randomly masked visual tokens, with reconstruction quality evaluated across mask ratios. The results indicate that visual tokens are highly redundant, with a small portion sufficient for holistic reconstruction while additional tokens recover finer details.
- MaskGIT reconstructs images by randomly masking visual tokens and applying its iterative decoding algorithm.The reconstruction setup varies the mask ratio r and evaluates PSNR and LPIPS as functions of r.
- PSNR and LPIPS measure reconstruction quality and diversity as functions of the masked-token ratio.
- Visual tokens are highly redundant in holistic image reconstruction.
- 10% of tokens are described as essential for holistic reconstruction, while remaining tokens improve finer appearance or details.
- The finding supports the masking design, in which predictions of the first few tokens are key to image generation.
B. Additional Class-conditional Image Generation Results
Additional ImageNet results evaluate MaskGIT's class-conditional generation quality and diversity against VQGAN, BigGAN, and ADM. MaskGIT achieves strong quality scores, improved recall, and state-of-the-art Inception Scores when using classifier-based rejection sampling.
- The additional experiments evaluate class-conditional image generation on ImageNet using classifier-based rejection sampling and feature-based Precision and Recall.
- 355.6 on 256x256 and 342.0 on 512x512 are MaskGIT's state-of-the-art Inception Scores with rejection sampling.
- MaskGIT consistently improves over VQGAN and is comparable with ADM using classifier guidance.
- MaskGIT's recall scores outperform those of BigGAN and VQGAN under both reported measures.
- MaskGIT outperforms BigGAN and baseline VQGAN by a large margin on CAS evaluated without RandAugment-trained classifier augmentation.
- The section includes comparisons with BigGAN-deep and VQVAE-2 for class-conditional samples.
C. Additional Examples of Class-conditional Image Editing Applications
The paper provides additional examples of class-conditional image editing and image-conditional panorama synthesis. These examples illustrate applications beyond standard image generation.
- Additional examples demonstrate class-conditional image editing applications.
- Additional examples demonstrate image-conditional panorama synthesis.
D. Image Outpainting Comparisons with SOTA Transformer-based Approaches
MaskGIT is compared with ImageGPT and VQGAN for image outpainting, using ground-truth context from either the top or bottom half. It supports higher-resolution outputs and outpainting in arbitrary directions, unlike the autoregressive baselines.
- MaskGIT and VQGAN achieve higher sample fidelity at higher resolutions than ImageGPT, which supports a maximum resolution of 192 x 192.Tokenization enables the higher-resolution operation of MaskGIT and VQGAN.
- MaskGIT can outpaint in arbitrary directions, whereas ImageGPT and VQGAN handle only one direction with a single model.The comparison uses extrapolation from either the top or bottom half of the ground-truth image.
E. Image Inpainting and Outpainting Comparisons with SOTA GAN-based Approaches
MaskGIT is evaluated against GAN-based image-completion methods on qualitative inpainting and outpainting examples, including challenging large-mask settings. Compared with prior GAN methods, it produces more coherent structures and fewer artifacts in the reported examples.
- The qualitative comparisons cover image completion on Places2 and report examples for both inpainting and outpainting.The section states that quantitative results were discussed separately.
- MaskGIT demonstrates stronger capability than prior GAN-based methods in completing structures coherently, with fewer artifacts in its samples.Examples include bridges and buildings that the compared GAN methods struggle to complete.
- MaskGIT is compared with CoModGAN under large masking ratios, conditioning on center regions of 50% x 50% and 31.25% x 31.25%.These settings are identified as challenging for traditional GANs.
F. Limitations and Failure Cases
The paper reports failure cases in outpainting and inpainting, including semantic or color shifts, boundary-object changes, and artifacts or oversmoothing in complex structures. These cases identify settings where further improvement remains necessary.
- Limited attention can cause semantic or color shifts when MaskGIT outpaints in opposite directions.The model may forget synthesized semantics or color from one end while generating the other.
- MaskGIT may ignore or modify objects located on image boundaries during outpainting and inpainting.
- Complex structures such as human faces, text, and symmetric objects can trigger oversmoothing or undesired artifacts.The paper identifies improvement for these circumstances as future work.