PaperlayerView exact revision on arXiv

Source-linked AI summary

Halton Scheduler For Masked Generative Image Transformer

Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, Matthieu Cord

arXiv:2503.17076v1cs.CV

TL;DR

MaskGIT’s scheduler has received less attention despite its role in token unmasking, and the Confidence scheduler can cluster token selections with low information gain. The paper analyzes mutual information in MaskGIT sampling and proposes a Halton-sequence scheduler that spreads tokens spatially. It reports improved quality and diversity on class-to-image and text-to-image generation, without additional noise injection or training.

  • Problem

    MaskGIT’s token-unmasking scheduler has received limited attention, although the Confidence scheduler can cluster selections and yield low information gain per step.

    Source: §1 · p. 2

  • Method

    The paper analyzes mutual information between sampled tokens and proposes a deterministic scheduler that uses the Halton low-discrepancy sequence to spread token selection spatially.

    Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 5

  • Results

    The Halton scheduler improves image quality and diversity in both class-to-image and text-to-image generation while keeping MaskGIT’s inference speed high.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The scheduler requires neither additional noise injection nor training and can be used as a drop-in replacement for the original sampling strategy.

    Source: §1 · p. 2, §5 · p. 9

    The scheme and MaskGIT remain unable to accommodate long-range dependencies.

    Source: §5 · p. 10

Abstract

from arXiv · show

Masked Generative Image Transformers (MaskGIT) have emerged as a scalable and efficient image generation framework, able to deliver high-quality visuals with low inference costs. However, MaskGIT's token unmasking scheduler, an essential component of the framework, has not received the attention it deserves. We analyze the sampling objective in MaskGIT, based on the mutual information between tokens, and elucidate its shortcomings. We then propose a new sampling strategy based on our Halton scheduler instead of the original Confidence scheduler. More precisely, our method selects the token's position according to a quasi-random, low-discrepancy Halton sequence. Intuitively, that method spreads the tokens spatially, progressively covering the image uniformly at each step. Our analysis shows that it allows reducing non-recoverable sampling errors, leading to simpler hyper-parameters tuning and better quality images. Our scheduler does not require retraining or noise injection and may serve as a simple drop-in replacement for the original sampling strategy. Evaluation of both class-to-image synthesis on ImageNet and text-to-image generation on the COCO dataset demonstrates that the Halton scheduler outperforms the Confidence scheduler quantitatively by reducing the FID and qualitatively by generating more diverse and more detailed images. Our code is at https://github.com/valeoai/Halton-MaskGIT.

1 INTRODUCTION

MaskGIT offers fast image inference, but its scheduler must balance parallel token unmasking against sampling errors. The paper proposes a Halton scheduler to spread tokens spatially, increase information gained per step, and improve image diversity and quality.

1 INTRODUCTION

  • Other iterative image-generation approaches, including reverse diffusion and next-token prediction, can produce photo-realistic images but impose slow, costly inference and require customized scheduling.The cited examples are reverse diffusion (Peebles & Xie, 2023; Esser et al., 2024) and next-token prediction (Yu et al., 2022).Source: §1 · p. 2
  • MaskGIT inference starts from a fully masked grid and iteratively unmasks scheduled tokens, aiming to unmask many per step while preventing parallel-sampling errors.MaskGIT represents images as discrete token grids and trains a network to predict masked token values.Source: §1 · p. 2
  • The Confidence scheduler can select clustered tokens around already predicted areas, yielding low information gain per sampling step and affecting image diversity and quality.It selects the most certain tokens first.Source: §1 · p. 2
  • The Halton scheduler uses spatially spread token selection to increase information per step and produce more diverse, higher-quality images.The paper evaluates it on class-to-image and text-to-image generation, reporting improved results without additional compute, data, or training constraints.Source: §1 · p. 2
  • The paper presents mutual-information analysis of MaskGIT sampling and evaluates the Halton scheduler on class-to-image and text-to-image generation.The scheduler requires no additional compute, data, or training constraints.Source: §1 · p. 2

2 RELATED WORK

Related work spans several image-generation families and MaskGIT extensions, while prior improvements to MaskGIT have often involved additional training or optimization. The paper situates its scheduler among work on deterministic sequences and focuses on MaskGIT’s sampling process.

2 RELATED WORK

  • GANs, diffusion, and autoregressive generation are distinct approaches to image synthesis; diffusion and autoregressive methods can produce high-quality outputs but have long inference times.The passage cites GANs (Radford et al., 2015; Brock et al., 2018; Karras et al., 2020; Kang et al., 2023), diffusion (Dhariwal & Nichol, 2021; Song et al., 2020; Rombach et al., 2022), and autoregressive generation (Yu et al., 2022; Ramesh et al., 2021; Ding et al., 2021).Source: §2 · p. 2
  • MaskGIT uses a bidirectional transformer encoder and masking strategy to reduce inference time, and its derivatives span text-to-image, text-to-video, point-cloud, and interactive-environment generation.The passage also describes MaskGIT as scalable, natively multimodal, and easy to integrate into large-scale self-supervised pipelines.Source: §2 · pp. 2–3
  • MaskGIT relies on a discrete visual tokenizer and cross-entropy training, while recent work has explored tokenizers for compression and semantic information.The passage contrasts the less extensively studied discrete tokenizers with continuous counterparts.Source: §2 · p. 3
  • QLDS uses a pseudo-random sequence for image compression, whereas this work focuses on improving MaskGIT’s generative capabilities.The paper investigates the Halton sequence for simplicity and efficiency; low-discrepancy sequences are deterministic and aim for uniform spatial coverage with minimal clustering or gaps.Source: §2 · pp. 3–4
  • Prior MaskGIT improvements have included tokenizer enhancement or post-hoc critic training, while FID-targeted optimization may lower FID without necessarily improving image quality.The cited critic approach requires additional parameters and retraining.Source: §2 · p. 3
  • The paper focuses on information gained at each sampling step and mutual information between tokens, without retraining or imposing a new tokenizer assumption.MaskGIT’s scheduler determines which tokens are unmasked at each step.Source: Figure 2 · p. 3, §2 · p. 4

3 METHOD

The section formulates MaskGIT scheduling through mutual information between tokens, then introduces a deterministic Halton sequence scheduler for spatially distributed unmasking. Compared with Confidence scheduling, Halton provides more uniform coverage and its performance keeps improving as inference steps increase.

3 METHOD

  • A schedule is an ordered partition of all tokens into non-overlapping sets, each set specifying which tokens are unmasked at one step.The method analyzes MaskGIT sampling through mutual information before introducing a Halton-based scheduler.Source: §3 · p. 4

3.1 THE ROLE OF MUTUAL INFORMATION

  • Because MaskGIT estimates token marginals rather than their joint, scheduling aims to reduce their gap, interpreted as mutual information, by limiting token uncertainty and dependence within each step.The analysis assumes conditional entropy between visual tokens is strongly correlated with their spatial distance.Source: §3.1 · pp. 4–5

3.2 LOW DISCREPANCY HALTON SEQUENCE

  • The Halton sequence uses quasi-random, spatially separated positions to cover the image uniformly; its 2D positions use bases 2 and 3 and are discretized to the token grid.The implementation discards duplicate grid coordinates and ensures all coordinates are sampled.Source: §3.2 · pp. 5–6

3.3 PROPERTIES OF SAMPLING METHODS

  • Confidence greedily selects low-entropy top-k tokens, clustering near revealed regions and neglecting token dependence; added selection noise reduces greediness but is difficult to tune as late-step entropy spikes.The noisy scheduler’s performance can saturate or deteriorate as the number of steps grows.Source: §3.3 · p. 6
  • Halton’s fixed token order distributes tokens across the image, balances the mutual-information terms, and lets performance keep improving as inference steps increase.The image composition is decided early, while later steps add high-frequency information.Source: Figure 3 · p. 5, §3.3 · p. 6
  • The analysis assumes the model accurately estimates each token’s marginal distribution and does not account for possible miscalibration or other biases.The authors report that their predictions remain consistent with experimental results despite this assumption.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments evaluate the Halton scheduler on class-to-image ImageNet and text-to-image COCO generation, comparing it with the Confidence scheduler and other methods. They report improved FID, scaling with more sampling steps, and qualitative image-quality benefits.

4.1 CLASS-TO-IMAGE SYNTHESIS

  • Halton improves FID by 2.19 on ImageNet 256×256 over Confidence and by 2.27 on ImageNet 512×512 using pretrained MaskGIT with 32 steps.The 256×256 result uses the authors’ MaskGIT model; the 512×512 result uses a public reproduction.Source: §4.1 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • Halton improves as sampling steps increase and outperforms Confidence above 12 steps, while Confidence performance deteriorates with more steps.The step-count comparison covers ImageNet 256×256 and 512×512.Source: §4.1 · p. 8, Figure 5 · p. 8
  • On ImageNet 256×256, the authors’ MaskGIT implementation with Halton reduces FID by almost 40% and improves IS by over 50% versus original MaskGIT.The comparison reports competitive performance among masked image modeling approaches without extensive optimization targeting FID.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 TEXT-TO-IMAGE SYNTHESIS

  • On zero-shot COCO, Halton reduces FID by 2.7 versus Confidence and by 10.1 versus aMused.The comparison reports that the scheduler’s quality gain over Confidence requires no additional computational cost.Source: §4.2 · p. 9, Table 4 · p. 9
  • Qualitative comparisons report that Halton-generated images have greater detail, diversity, sharpness, and background variety than Confidence-generated images.The section also presents curated class-to-image and text-to-image examples.Source: §4.2 · p. 9

5 CONCLUSION

The conclusion presents Halton as a scheduler that improves MaskGIT image quality and diversity without noise injection or retraining. It identifies long-range dependencies as a limitation shared by the scheme and MaskGIT.

5 CONCLUSION

  • Halton improves the quality and diversity of class-to-image and text-to-image generation while retaining MaskGIT’s inference speed.The scheduler spreads selected tokens spatially using a low-discrepancy Halton sequence and needs neither additional noise injection nor training.Source: §5 · p. 9
  • Curated examples show generated class-to-image and text-to-image samples across classes and diverse textual prompts.The text-to-image prompts and additional randomly selected class-to-image samples are available in the Appendix.Source: Figure 6 · p. 10, Figure 7 · p. 10
  • The scheme and MaskGIT cannot accommodate long-range dependencies; addressing them while retaining MaskGIT’s inference speed remains an open research frontier.The conclusion suggests that this may require rethinking image encoding, scheduling, and decoding as an integrated framework.Source: §5 · p. 10

A TRAINING DETAILS

The training-details section points to tables for the models’ hyperparameters and architectures, then describes the class-to-image and text-to-image configurations.

A TRAINING DETAILS

  • Table 5 lists training hyperparameters for both modalities, and Table 6 details their model architectures.Further class-to-image training configurations are located in the Appendix.Source: A TRAINING DETAILS · p. 15, Table 5 · p. 15, Table 6 · p. 15
  • Class-to-image models use a DiT-XL-like architecture with patch size 2; GPU memory constraints led the authors not to use EMA.The patch size reduces the token grid from 32×32 to 16×16, and tied input/output embeddings reduce trainable parameters.Source: A TRAINING DETAILS · p. 15
  • Text-to-image uses T5-XL embeddings for 120 text tokens and a DiT-L-like model with cross-attention for text conditioning.The model was the largest they could fit on their GPU with EMA.Source: A TRAINING DETAILS · p. 15

B EUCLIDEAN DISTANCE IN THE TOKEN SPACE

The section assumes that spatially close image tokens tend to have similar appearances and examines that relationship in token space. Its figure maps each token’s appearance dissimilarity from a reference token.

B EUCLIDEAN DISTANCE IN THE TOKEN SPACE

  • The analysis assumes nearby image parts tend to look alike, a relationship the authors say also holds in token space.They measure appearance dissimilarity using Euclidean distances between LlamaGen tokenizer latent representations on ImageNet.Source: B EUCLIDEAN DISTANCE IN THE TOKEN SPACE · pp. 15–16
  • Figure 8 colors tokens by normalized appearance dissimilarity from a red reference token; dark blue marks closest, most similar tokens and yellow marks more distant, dissimilar ones.The prose reports that spatially close tokens have closer feature representations, while those farther away tend to be dissimilar.Source: Figure 8 · p. 16, B EUCLIDEAN DISTANCE IN THE TOKEN SPACE · p. 16

C GENERATIVE METHOD COMPARISON

The comparison places Halton-scheduled masked image modeling closer to diffusion and autoregressive methods while retaining its speed advantage.

C GENERATIVE METHOD COMPARISON

  • Halton scheduling narrows the gap between diffusion, autoregressive, and masked image modeling methods while retaining dozens of inference steps versus hundreds for the other methods.Source: C GENERATIVE METHOD COMPARISON · p. 16

D INTERMEDIATE GENERATION

The section compares Halton with other class-conditional methods, examines intermediate generations, and describes the sequence used to order token unmasking.

D INTERMEDIATE GENERATION

  • 25/1024 (~2%) unmasked tokens set the bird’s color and background after four steps; shape appears fixed by step eight, and texture by step twelve.At 12 steps, only 16% of tokens are predicted; the last 12 steps lower FID by just 0.14 points, and later steps may be skipped as a quality–compute compromise.Source: D INTERMEDIATE GENERATION · p. 16, Figure 10 · p. 18, Table 8 · p. 18
  • Halton outperforms the original MaskGIT and is competitive among masked image modeling approaches.The comparison with other methods is summarized in a radar plot whose metrics are normalized to each metric’s best model, with FID reversed so higher is better.Source: Table 7 · p. 17, Figure 9 · p. 17

E PSEUDO-CODE FOR HALTON SEQUENCE

  • Two Halton sequences with bases 2 and 3 define 2D coordinates that are discretized onto a 32 × 32 grid to order token unmasking.Duplicate points are discarded, and the sequence length is set to ensure complete grid coverage.Source: E PSEUDO-CODE FOR HALTON SEQUENCE · p. 18

F TEXT PROMPTS

  • Random class-to-image samples from Halton show greater detail than Confidence samples, particularly in background features.Source: Figure 11 · p. 20

G RANDOM SAMPLES FROM OUR CLASS CONDITIONED MODEL

The class-conditioned sample comparison reports richer detail and diversity with Halton than with Confidence, especially in backgrounds.

G RANDOM SAMPLES FROM OUR CLASS CONDITIONED MODEL

  • Halton produces more diverse images and intricate details than Confidence, whose samples are often smooth with poorly defined backgrounds.The diversity difference is particularly evident in background elements.Source: G RANDOM SAMPLES FROM OUR CLASS… · p. 20

H FAILURE CASES

The failure cases include duplicate objects, irreversible token errors, and difficulty with complex classes or prompts.

H FAILURE CASES

  • Early spatial spreading can yield multiple instances of an object or repeated parts of one entity, such as a two-headed bird.Source: H FAILURE CASES · pp. 20–21
  • Long-range correlations remain a challenge for MaskGIT with the Halton scheduler.Source: Figure 12 · p. 21
  • Once a token is predicted, it remains fixed even if incorrect, so MaskGIT cannot correct that prediction during generation.Source: H FAILURE CASES · p. 21
  • The model struggles to generate human faces or bodies accurately for some ImageNet classes.Source: H FAILURE CASES · p. 21
  • For intricate or abstract text prompts, the model can fail to compose coherent scenes or render text faithfully.Source: H FAILURE CASES · p. 21
Loading 2503.17076v1…