Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens

1. Introduction

• The work asks whether efficient, effective text-to-image generation can use only open data, amid high training and inference costs and reliance on proprietary datasets.It identifies TiTok’s two-stage training, VQ-only representation, and focus on low-level details as barriers to scaling and textual alignment.Source: §1 · pp. 1–2
• Building on compact 1D tokenization, which removes fixed patch-to-token correspondence, the paper develops a tokenizer and generator for text-to-image models.The introduction contrasts this with 2D grid-based tokenizers and notes the sampling-speed benefits attributed to TiTok.Source: §1 · p. 1
• TA-TiTok uses one-stage training instead of TiTok’s complex two-stage pipeline, supporting scalable training on large text-image datasets.The paper presents this as removing multi-stage complexity.Source: §1 · p. 2
• TA-TiTok extends 1D tokens to continuous VAE representations, avoiding VQ quantization loss while combining compact-token sampling efficiency with improved reconstruction quality.The paper reports more consistent and accurate reconstructions than the VQ counterpart.Source: §1 · p. 2
• TA-TiTok incorporates caption text during de-tokenization by concatenating CLIP embeddings with image representations to improve semantic alignment and retain fine details.The tokenizer is trained on a large-scale dataset such as DataComp.Source: §1 · p. 2
• MaskGen supports discrete tokens trained with cross-entropy and continuous tokens trained with diffusion loss, using CLIP conditioning and separate adaptive LayerNorm parameters for text and image modalities.The work uses publicly available and synthetic image sources, filtering by aesthetic score, resolution, and watermarks, and recaptions selected subsets.Source: §1 · pp. 2–3
• On MJHQ-30K, MaskGen-L (568M) achieves generation FID 7.74 versus Show-o’s 14.99, with 30.3× faster inference throughput.The passage also reports that MaskGen-L surpasses SD-2.1 (26.96) and PixArt-α (9.85), using 2% and 21% of their training times, respectively; MaskGen-XL reports FID 7.51 and 6.53 on MJHQ-30K and GenEval overall scores 0.57 and 0.55 with discrete and continuous tokens, respectively.Source: §1 · p. 3
• The authors plan to release TA-TiTok and MaskGen training code and weights to promote research and access to text-to-image masked generative models.They describe MaskGen as the first open-weight, open-data model in this category with performance comparable to state-of-the-art models.Source: §1 · p. 3

Image Tokenization

• Discrete tokenization maps tokens to codebook entries and is suited to autoregressive or masked generation, while continuous VAE tokenization uses normally distributed latent representations.The passage notes that continuous tokenization was adapted to masked generation in MAR using a diffusion module to sample tokens.Source: §2 · p. 3

Image Generation with Sequence Models

• Autoregressive models predict image tokens sequentially, whereas masked generative models predict masked tokens simultaneously, giving masked models a sampling-speed advantage.The passage describes this contrast as GPT-like versus BERT-like generation.Source: §2 · p. 3

Text-to-Image Generation

• Text-to-image work includes diffusion, masked generative, and autoregressive models, with recent diffusion advances identified as potential improvements for masked generation.Muse and Parti are cited as examples of successful masked generative and autoregressive text-to-image approaches.Source: §2 · p. 3

3. Preliminary

• TiTok retains latent-token embeddings from a Vision Transformer encoder as a compact 1D representation, then quantizes them to nearest codebook entries.The image patches are concatenated with latent tokens before encoding.Source: §3 · p. 3
• TiTok reconstructs images by passing quantized codes concatenated with mask tokens through a Vision Transformer decoder.The displayed formulation specifies sequence concatenation for both encoding and decoding.Source: §3 · pp. 3–4
• Figure 2 presents TA-TiTok’s one-stage training, discrete and continuous token options, and text-guided de-tokenization; its reconstructions surpass TiTok’s.Source: Figure 2 · p. 4
• Discrete masked generative models mask image tokens and train a bidirectional transformer to predict codebook tokens using cross-entropy loss.At sampling, masked positions are iteratively predicted, while uncertain positions can be re-masked.Source: §3 · p. 4
• Continuous masked generative models use diffusion loss to approximate token distributions, with predicted tokens masked randomly at each iteration.A transformer supplies conditioning vectors to an MLP denoiser for each masked token.Source: §3 · p. 4

4.1. TA-TiTok: Text-aware 1D Tokenizer

• TA-TiTok’s one-stage recipe uses ResNet50 perceptual features and strengthens PatchGAN with blur kernels and LeCAM regularization for 1D-token reconstruction.The paper reports that these enhancements improve image reconstruction quality through 1D tokens.Source: §4.1 · p. 4
• The continuous variant models 1D latents as a Gaussian with KL regularization rather than quantizing them, avoiding quantization loss while retaining compact-token efficiency.The KL variant is described as a drop-in replacement for standard 2D VAEs in diffusion models.Source: §4.1 · p. 5
• TA-TiTok concatenates projected CLIP text embeddings with image and mask tokens in its decoder; decoder-only text input showed similar performance to encoder-and-decoder input.The sequence adds 77 CLIP text tokens to 32, 64, or 128 latent tokens and requires fewer computations than typical 2D tokenizers using 256 tokens.Source: §4.1 · p. 5

4.2. MaskGen: T2I Masked Generative Model

• MaskGen combines CLIP text embeddings with TA-TiTok tokens in MM-DiT blocks, uses separate adaptive LayerNorms for text and image, and conditions on aesthetic scores.CLIP is selected over T5-XXL for computational efficiency and reduced storage demands.Source: §4.2 · p. 5, Figure 3 · p. 6
• MaskGen trains discrete tokens with cross-entropy and continuous tokens with diffusion loss, while compact 1D sequences reduce token counts for training and sampling.The continuous-token diffusion loss uses an MLP to approximate each masked token’s distribution.Source: §4.2 · pp. 5–6
• Table 1 covers two MaskGen configurations that scale up MM-DiT blocks.Source: Table 1 · p. 6

5.1. Implementation Details

• Experiments use TA-TiTok variants with 32, 64, or 128 tokens and MaskGen-L/XL, trained on open datasets and evaluated on ImageNet, MJHQ, and GenEval.The evaluation uses reconstruction metrics for TA-TiTok and FID and GenEval scores for MaskGen generation.Source: §5.1 · p. 6

5.2. Optimized Image Tokenization with TA-TiTok

• 2.72 rFID improvement over the original schemes [74] is reported for the improved one-stage training recipe.Attention maps show latent tokens focusing on meaningful semantic regions, while code swaps change corresponding entities or, for latent 11, overall image style.Source: §5.2 · p. 7, Table 2 · p. 7, Figure 4 · p. 7
• TA-TiTok outperforms non-text-aware TiTok across token-count and tokenizer configurations, with the largest gains for fewer tokens and discrete tokens.Continuous tokens consistently outperform discrete tokens; the authors attribute TA-TiTok’s larger gains in low-token and VQ settings to text embeddings supplementing semantic information.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Text-to-Image Generation with MaskGen

• On MJHQ-30K, discrete MaskGen-XL reaches FID 7.51 vs. LlamaGen’s 25.59 and Show-o’s 14.99; continuous MaskGen-XL reaches 6.53 vs. SDXL’s 8.76.MaskGen-L with continuous tokens reaches FID 7.24 vs. PixArt-α’s 9.85, with 1.4× faster inference and less than one-fifth of its training resources.Source: §5.3 · pp. 7–8, Table 4 · p. 8
• MaskGen-XL achieves a GenEval overall score of 0.57 with discrete tokens; its continuous-token version scores 0.55, comparable to recent diffusion models.MaskGen-L with discrete tokens scores 0.53, outperforming LlamaGen by 0.21 and performing on par with the larger Show-o.Source: §5.3 · p. 8, Table 5 · p. 8
• Ablations report that TA-TiTok and aesthetic-score conditioning each enhance MaskGen generation quality, with the tokenizer’s impact described as larger.The comparison uses a continuous-token version of MaskGen-L.Source: Table 6 · p. 8, §5.3 · p. 8

• TA-TiTok provides text-aware 1D tokenization, while MaskGen supports discrete and continuous tokens; compact tokenization lowers training costs and accelerates sampling.The authors state that public-data training broadens access and that code and model weights are released.Source: §6 · p. 8

Appendix

• The appendix maps supplementary sections covering implementation, tokenizer and MaskGen ablations, token-type comparisons, extra COCO results, qualitative examples, and limitations.It identifies sections A through I and their topics.Source: Appendix · p. 12

A. More Implementation Details

• Training data preparation applies resolution filtering across both models, plus aesthetic and watermark filtering for MaskGen; synthetic datasets are exempted.Noisy web-caption datasets are recaptioned with Molmo, with augmented and original captions sampled at a 95:5 ratio during training.Source: A. More Implementation Details · p. 12
• TA-TiTok uses batch size 1024 for one epoch of 650k steps, while MaskGen’s discrete and continuous variants use different batch sizes and learning-rate schedules.The appendix specifies batch sizes of 4096 for discrete MaskGen and 2048 for continuous MaskGen, with cosine and constant learning-rate schedules, respectively.Source: A. More Implementation Details · p. 12

B. Ablation Studies for TA-TiTok

• CLIP text-encoder embeddings yield a marginal improvement over numerical IDs from the CLIP text tokenizer.The comparison is an ablation of text-guidance type in TA-TiTok.Source: B. Ablation Studies for TA-TiTok · p. 12, Table 9 · p. 13
• The training-data table describes resolution, aesthetic-score, and watermark filtering, and reports remaining text-image pairs after filtering.It also notes Molmo recaptioning for datasets with noisy web-crawled captions.Source: Table 7 · p. 13
• The recaptioning prompts replace {original_caption} with the image’s original caption and request a concise, detailed English revision.The figure caption says one of four prompts is used for each image.Source: Figure 5 · p. 13, B. Ablation Studies for TA-TiTok · p. 13
• Adding text guidance to both encoder and decoder has negligible reconstruction impact, so decoder-only guidance provides similar quality with a simpler structure.The evaluation uses zero-shot ImageNet reconstruction metrics.Source: B. Ablation Studies for TA-TiTok · pp. 12–13, Table 10

C. Ablation Studies for MaskGen

• Captions augmented by Molmo provide richer details and improved alignment with image content.Source: Figure 6 · p. 14
• More tokens improve generation quality but increase training time and slow inference.The ablation evaluates FID on MJHQ-30K and overall GenEval score.Source: C. Ablation Studies for MaskGen · pp. 13–14, Table 11 · p. 14
• Higher aesthetic-score conditioning is associated with more dramatic lighting and fine-grained details, offering control over generated images.The visual comparison holds prompts and other hyperparameters constant.Source: C. Ablation Studies for MaskGen · p. 14, Figure 7 · p. 15

D. Comparisons Between MaskGen Using Discrete and Continuous Tokens

• KL performs better on MJHQ-30K FID, while VQ scores higher on GenEval; KL's diverse, high-aesthetic images and VQ's compositional accuracy distinguish their strengths.The authors hypothesize that KL's diversity contributes to its FID result; VQ is constrained by a finite codebook.Source: D. Comparisons Between MaskGen Using… · p. 14
• The VQ variant trains faster and infers significantly faster than KL; using 128 VQ tokens and 32 KL tokens keeps training costs comparable.The passage attributes the efficiency difference primarily to differences in KL's diffusion process.Source: D. Comparisons Between MaskGen Using… · p. 14

E. Zero-Shot Text-to-Image Generation Results on COCO

• 8.98 FID-30K for MaskGen-XL and 9.66 FID-30K for MaskGen-L; the authors report comparable zero-shot performance to state-of-the-art models.Evaluation uses 30K image-caption pairs from COCO 2014 validation and the pretraining-stage model.Source: E. Zero-Shot Text-to-Image Generation… · p. 15

F. KL variant of TiTok on ImageNet

• 3.8× training-time reduction and 8.1× inference speedup with 64 tokens; 2.7× and 3.2×, respectively, with 128 tokens.MAR with the KL TiTok variant maintains performance comparable to MAR with conventional 2D VAEs using 256 tokens.Source: F. KL variant of TiTok on ImageNet · p. 15

G. Qualitative Examples of Latent Code Swapping

• VQ latent-token swapping changes the appearance of the corresponding semantic entity, and tokens capture meaningful image elements without requiring the generator.The passage links this semantic encoding to efficient 1D tokenization and competitive reconstruction fidelity.Source: G. Qualitative Examples of Latent Code… · p. 15

H. Qualitative Examples of MaskGen

• MaskGen's qualitative examples illustrate high-fidelity and diverse text-to-image generations using TA-TiTok.Source: H. Qualitative Examples of MaskGen · p. 15

I. Limitations and Future Work

• MaskGen-XL is limited to 1.1B parameters by computational resources, and increasing KL token count raises training costs through longer convergence.The current KL variant is designed for 32 tokens; more tokens improve tokenization quality and reconstructed samples.Source: §I · p. 15
• MaskGen currently operates at 256 × 256, while high-resolution implementations such as Muse validate scalability of the core 1D-tokenization and masked-generation design.Source: §I · p. 15
• Future work will focus on optimizing convergence speed, scaling up the model, and enabling high-resolution outputs.Source: §I · p. 15