Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model

1 INTRODUCTION

• Autoregressive decoding generates image tokens sequentially, making inference slow and rigid; AR-language-model-plus-diffusion hybrids still lack a shared generative paradigm.Generating one image can require thousands of token predictions, each triggering a full network forward; the fixed order also limits flexible conditioning such as inpainting without fine-tuning.Source: §1 · p. 1
• Dual-Diffusion and Diffuse Everything claim discrete-diffusion unification but use continuous diffusion for images, while UniDisc applies discrete diffusion across modalities [Li et al., 2024c; Rojas et al., 2025; Swerdlow et al., 2025a].UniDisc struggles with 1024×1024 images, does not match Stable Diffusion 1.5 fidelity, and lacks VQA support.Source: §1 · pp. 1–2
• Existing unified discrete-diffusion models are typically trained from scratch on mixed-modality tokens, limiting generative fidelity and transferability without rich pixel-level priors.The paper identifies this lack of strong pretrained discrete-diffusion backbones as a generalization and scalability bottleneck.Source: §1 · p. 2
• Muddit combines parallel discrete diffusion with a pretrained Meissonic image prior and lightweight text decoder, and reports superior performance and efficiency across multimodal benchmarks.The evaluations include GenEval, CIDEr, VQAv2, MME, and GQA.Source: §1 · p. 2

2.1 DISCRETE DIFFUSION WITH UNIFIED IMAGE AND TEXT PERSPECTIVE

• Discrete diffusion corrupts tokens toward a mask state and reverses the process by updating masked positions from categorical predictions, allowing the same process across discrete text and image alphabets.The reverse posterior leaves observed tokens unchanged and, for masked tokens, combines the mask and clean token according to survival probabilities.Source: §2.1 · pp. 2–3

2.2.1 UNIFIED ARCHITECTURE

• Muddit uses a single MM-DiT generator initialized from the pretrained Meissonic image model, while discrete encoders and decoders map image and text data into and out of compact token spaces.The image encoder is a pretrained VQ-VAE, and a lightweight linear text decoder converts predicted tokens back to text.Source: §2.2.1 · pp. 3–4

2.2.2 UNIFIED TRAINING

• Training randomly masks either modality and optimizes one masked-token predictor with the same objective for text-to-image and image-to-text generation.The conditioning input changes between directions, while the loss remains unchanged; mask ratios are sampled for each modality before corruption.Source: §2.2.2 · pp. 4–5

2.2.3 UNIFIED INFERENCE

• One generator and sampler support text-to-image, captioning, and VQA by changing the condition; reverse diffusion iteratively updates masked tokens, predicting multiple positions in parallel.For VQA, the question is followed by mask tokens that are predicted as the answer.Source: Figure 2 · p. 4, §2.2.3 · p. 5

3.1 EXPERIMENTAL SETUP

• Experiments initialize the MM-DiT from Meissonic, add a lightweight text decoder, and train on filtered image–text data through pretraining and instruction tuning.The reported setup includes roughly 10 million image–text pairs; the described pretraining and instruction-tuning stages use 8 million and 1 million samples, respectively, plus 1 million curated pairs.Source: §3.1 · p. 6

3.2 TEXT-TO-IMAGE GENERATION

• After supervised fine-tuning, Muddit's 512 × 512 model attains 0.61 overall accuracy on GenEval, exceeding Monetico (0.44) and Meissonic (0.54), and closely matching Stable Diffusion 3 (0.62) with 1B parameters.Source: §3.2 · p. 6

3.3 IMAGE-TO-TEXT GENERATION

• On MS-COCO, Muddit scores 59.9 CIDEr versus D-DiT’s 56.2; it reaches 68.2% accuracy on VQAv2, 1107.4 on MME, and 57.5 on GQA.The authors report that its VQAv2 accuracy also exceeds Show-O and D-DiT.Source: §3.3 · p. 7, Table 2 · p. 9
• For image captioning and VQA, examples show visual grounding and detailed descriptions alongside answers to questions about counting, color, materials, and composition.The captioning examples span humans, animals, vehicles, and natural landscapes.Source: Figure 4 · p. 7, §3.3 · p. 8, Figure 5 · p. 8

3.4 ABLATION STUDY AND ANALYSIS

• Performance generally plateaus around T = 32: GenEval and CIDEr improve substantially from T = 8 to T = 32, while VQAv2 remains stable across timesteps.The authors identify a moderate step count as a balance between accuracy and efficiency.Source: §3.4 · pp. 8–9, Table 5 · p. 9
• Text-loss weights around 0.6 yield the best overall performance: CIDEr and GenEval peak near that value, while VQAv2 improves with stronger text supervision before plateauing.The same text-loss weight is applied during pretraining and instruction tuning.Source: §3.4 · p. 9, Table 3 · p. 9
• Joint training yields the highest GenEval score; I2T-only training drops GenEval from 61.6 to 28.3, while CIDEr remains nearly unchanged and VQAv2 declines only slightly.The authors conclude that separating objectives weakens cross-modal integration.Source: §3.4 · p. 9, Table 4 · p. 9

3.5 THE SCALABILITY OF MUDDIT

• After scaling training, Muddit improves consistently across image generation and understanding benchmarks and outperforms similar-sized hybrid models despite using substantially less data.The authors attribute this data efficiency to the pretrained visual prior and unified mask-token prediction across tasks, rather than hybrid models’ combination of prediction objectives and additional special tokens.Source: §3.5 · pp. 9–10, Table 7 · p. 10

4 CONCLUSION

• Muddit reports strong performance across text-to-image, image-to-text, and VQA, matching or outperforming significantly larger autoregressive models while enabling fast, parallel inference.The conclusion presents discrete diffusion as a general-purpose modeling strategy and a potential scalable backbone for future multimodal systems equipped with strong visual priors.Source: §4 · p. 10

APPENDIX OVERVIEW

• The appendix adds qualitative and experimental results, ablations, inference-time analysis, reverse-diffusion examples, a discussion of limitations and broader impacts, and an account of LLM use.It also reviews related work on unified multimodal models and masked image modeling.Source: APPENDIX OVERVIEW · p. 17

A.1 UNIFIED MODELS FOR GENERATION AND UNDERSTANDING

• Unified-generation approaches include fully autoregressive models with sequential-decoding latency, split text-AR/image-diffusion systems, and fully discrete diffusion models that support parallel sampling but lag in fidelity and scale.The review also notes that GPT-4o's closed-source status obscures architectural and training details, while its success may reflect scale rather than architectural novelty.Source: §A.1 · pp. 17–18

A.2 MASKED IMAGE MODELING

• Masked image modeling trains models to predict or reconstruct obscured image content from visible context, using pixels, latent patches, or discrete tokens.MaskGIT introduced parallel decoding through iterative token refinement; RandomAR and MAR differ from it in inference scanning order.Source: §A.2 · p. 18
• The paper situates masked-image techniques as a conceptual foundation for discrete diffusion over tokenized spaces and as important to modern unified models.The paper says it will introduce discrete diffusion in the following section.Source: §A.2 · p. 18

A.3 RELATIONSHIP TO CONCURRENT WORK

• Muddit is presented as a unified, visual-prior fully discrete diffusion model intended for image understanding as well as text-to-image generation.The authors describe a visual-first backbone as a practical, competitive alternative to the more common LLM-first unified paradigm.Source: §A.3 · p. 18
• Prior unified discrete diffusion models such as UniDisc were trained from scratch on multimodal data and lacked strong visual priors.The passage says they underperformed early diffusion baselines and did not support VQA tasks.Source: §A.3 · p. 18
• Muddit is described as the first unified discrete diffusion model built on a pretrained high-resolution text-to-image backbone with a lightweight text decoder.The authors state that this visual prior improves discrete diffusion's scalability and generalization behavior through a visual backbone aligned with text.Source: §A.3 · p. 18

B ADDITIONAL QUALITATIVE RESULTS

• The appendix supplies further image-to-text and text-to-image examples.The passages identify these as additional visualizations without describing specific outcomes.Source: B ADDITIONAL QUALITATIVE RESULTS · p. 18
• Muddit's VQA examples identify fine-grained attributes and object categories, answer an affordance question, and handle commonsense reasoning and spatial localization.Examples include identifying blonde hair or a beagle and judging whether crossing at a red light is legal.Source: B ADDITIONAL QUALITATIVE RESULTS · p. 18
• For image-guided text editing, Muddit completes masked sentences from an input image with phrases described as semantically and visually grounded.The passage frames this as a task where the model fills blanks in a partially masked caption conditioned on the image.Source: B ADDITIONAL QUALITATIVE RESULTS · p. 18

C ADDITIONAL EXPERIMENTAL RESULTS

• Muddit demonstrates strong performance in existence, color, and scene understanding on the MME benchmark.The detailed results are presented in Table 8.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 19, Table 8 · p. 19
• The MME breakdown also reports solid reasoning capabilities for Muddit.The passage does not provide a specific score in this excerpt.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 19
• Table 8 is identified as the detailed MME results table.Its caption establishes the table's scope, but not individual values.Source: Table 8 · p. 19

• Moderate CFG values, such as 1.5, yield the best results on MS-COCO captioning and VQAv2.Source: §D.1 · p. 19
• Higher CFG scales lead to degraded performance on the reported benchmarks.Source: §D.1 · p. 19
• The ablation reports CFG-scale performance for MS-COCO captioning and VQAv2.Source: §D.1 · p. 19

• 1.49 seconds average latency; Muddit is 4× to 11× faster than competitive baselines, including Qwen-2.5-VL, Show-o, BLIP-2, and LLaVA-1.6.The reported speedups are 4.2×, 5.6×, 8.1×, and 10.9×, respectively.Source: E INFERENCE TIME ANALYSIS · p. 19
• Without KV caching, AR attention costs O(t^2D) per step; with caching, it costs O(tD) while attending to t stored keys.Source: E INFERENCE TIME ANALYSIS · pp. 19–20
• Table 10 reports Muddit’s best overall balance: highest text-to-image throughput matched with significantly faster image-to-text throughput than other models.The comparison covers images per second and tokens per second across resolutions and steps.Source: Table 10 · p. 20, E INFERENCE TIME ANALYSIS · p. 20
• Discrete diffusion updates the full sequence in parallel over T iterations, offering an L/T speedup over autoregressive decoding without caching; practical throughput is often competitive or faster.Against AR with KV caching, discrete diffusion may appear less efficient in theoretical FLOPs.Source: E INFERENCE TIME ANALYSIS · p. 20

F GENERATED RESULTS STEP BY STEP

• Muddit starts text generation from a fully masked sequence of 77 token indices and runs 16–32 denoising steps.A shared transformer predicts categorical distributions over all positions in parallel at each step, and a sampler selects the next sequence.Source: F GENERATED RESULTS STEP BY STEP · p. 20
• At each reverse-diffusion step, the sampler uses the transformer’s predictions and the current sequence to produce the next sequence, conditioned on c.The resulting sequence seeds the following step; logits may be tempered or top-k filtered before independent token sampling.Source: F GENERATED RESULTS STEP BY STEP · p. 20
• Parallel updates preserve global syntactic and semantic structure, and 16–32 steps are reported sufficient to approximate the natural-language distribution with high fidelity.Source: F GENERATED RESULTS STEP BY STEP · p. 21

G.1 LIMITATIONS

• Token-level discrete representations may underperform continuous diffusion for photorealistic or high-resolution image generation.The pretrained text-to-image foundation also limits rich text understanding and generation relative to the latest large language models, making Muddit less suitable for long-form tasks or deep linguistic reasoning.Source: §G.1 · p. 21

G.2 BROADER IMPACTS

• Muddit offers a complementary path toward efficient, grounded multimodal generation, particularly in vision-centric applications.The paper identifies masked captioning, image editing, and code implementation as possible downstream completion tasks.Source: §G.2 · p. 21
• Synthetic content creation with generative models carries a risk of misuse.Source: §G.2 · p. 21

H THE USE OF LARGE LANGUAGE MODELS

• The authors report that large language models were used only for language polishing and minor editing; research ideas, methods, and experiments were carried out by the human authors.Source: H THE USE OF LARGE LANGUAGE MODELS · p. 21

Muddit

• Figures present Muddit examples for image-to-text, text-to-image, visual question answering, and image-guided text editing.Source: Figure 6 · p. 22, Figure 7 · p. 23, Figure 8 · p. 24
• Figures 10 and 11 show image-to-text generated results at each step.Source: Figure 10 · p. 25, Figure 11 · p. 26
• The inference-speed comparison uses 32 Muddit inference steps and fixes sequence length at 77 across all models.Source: Figure 12 · p. 27