PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes

Chuyan Chen, Haoxing Chen, Kun Chen, Zhenglin Cheng, Long Cui, Ruishan Fang, Zhangxuan Gu, Zhicheng Huang, Zhenzhong Lan, Yuanting Lei, Haoquan Li, Jianguo Li, Rongchuan Li, Sidu Li, Tao Lin, Deyuan Liu, Jiacheng Liu, Lin Liu, Yuxuan Lou, Zhisheng Lu, Yuxin Ma, Shuheng Shen, Peng Sun, Chaoyang Wang, Hongjun Wang, Xiaomei Wang, Yongxin Wang, Chengzhang Wu, Hongru Wu, Jun Xie

arXiv:2609.03796v1cs.CVcs.AI

TL;DR

Open image-generation systems need broad visual capabilities together with attainable data, stable training, and efficient deployment, but conventional recipes rely heavily on paired captions early. LLaDA-Image separates visual-prior learning from language alignment, unifies understanding, generation, and editing, and distills the result for fast inference; it reports leading open-source Qwen-Image-Bench scores and releases weights, code, and recipes.

  • Problem

    Open visual creation systems must combine broad capabilities with attainable data budgets, stable large-scale training, and efficient deployment, while early paired-caption training is expensive and lossy.

  • Method

    LLaDA-Image trains a 6B Diffusion Transformer from scratch using image-only pre-training and mid-training, a frozen dLLM-based VLM with RQA–connector conditioning, progressive generation–editing training, and TwinFlow distillation.

  • Results

    53.53 and 53.38 overall on Qwen-Image-Bench English and Chinese tracks, respectively, exceeding Z-Image Turbo by 1.87 and 0.67 points; Turbo requires 2–4 sampling steps.

  • Takeaways & Limitations

    The released model family provides full multi-step high-fidelity generation and unified editing alongside a lower-cost 2–4-step deployment variant, with open weights, code, and recipes.

  • Takeaways & Limitations

    Remaining challenges include broader world knowledge, reliable long and multi-region text rendering, and native 2K generation, which remains unevaluated.

Abstract

from arXiv · show

We introduce LLaDA-Image, a unified framework that pairs a 6B Diffusion Transformer (DiT) trained from scratch with a frozen vision-language understanding module built on the LLaDA2.0-Mini diffusion language model backbone. Instead of relying heavily on paired image-text data from the beginning, we first build a strong visual generative prior through image-only pre-training and mid-training. The generation pipeline comprises 220M samples, 98 of which are real images. For efficient and scalable optimization, we use parameter-free RMSNorm throughout the DiT together with the Muon optimizer. The resulting unified model produces highly photorealistic images while accurately following fine-grained editing instructions. We further distill LLaDA-Image into LLaDA-Image-Turbo, enabling fast inference in 2-4 sampling steps. On Qwen-Image-Bench, LLaDA-Image achieves overall scores of 53.53 and 53.38 on the English and Chinese tracks, respectively, setting a new state-of-the-art among open-source models on both tracks. To support further research on capable and efficient generative models, we release our model weights, training code, and detailed recipes.

1 Introduction

LLaDA-Image addresses open visual creation by separating visual-prior learning from language alignment and combining multimodal understanding, generation, editing, and efficient deployment in one released framework. Its recipe emphasizes image-only and real-data-dominant training, unified conditioning, and publicly reproducible checkpoints and code.

  • Open visual creation systems must combine photorealism, instruction understanding, reference preservation, multilingual text rendering, attainable data budgets, stable training, and efficient deployment.
  • Paired captions are expensive and lossy, while synthetic-heavy training can propagate artifacts and limit long-run realism.
  • LLaDA-Image is a 6B-parameter Diffusion Transformer trained from scratch, progressively combining image-only visual-prior learning, language alignment, refinement, and joint generation–editing training.
  • The unified design uses a dLLM-based vision-language model, Residual Query Adapter, connector, and single-stream DiT to support understanding, generation, and editing.
  • The progressive curriculum moves from 256^2 image-only pretraining through aspect-ratio mid-training, paired alignment, refinement, and joint generation–editing training, while keeping real images above 70% during supervised fine-tuning.
  • TwinFlow distills the original model into LLaDA-Image Turbo variants requiring only 2–4 sampling steps.
  • The project releases Base and Turbo checkpoints, FP8 variants, training and inference code, and detailed recipes for reproducible model development.

2 Model Design

LLaDA-Image combines a dLLM-based VLM, connector, and single-stream DiT into one framework for understanding, generation, and editing. Reference images use a separate DiT pathway that combines semantic and pixel-level signals.

  • The architecture uses a dLLM-based VLM, an understanding-to-generation connector, and a DiT for image synthesis.The VLM handles multimodal understanding, the connector projects representations, and the DiT performs synthesis.
  • The VLM processes prompts and self-conditioning visuals, while editing reference images bypass it and enter through a dedicated DiT pathway.This separates instruction understanding from direct reference-image conditioning.
  • RQA and a Transformer Connector extract generation-relevant VLM representations and align them with the DiT conditioning space.RQA elicits generation-centric information; the connector translates it into DiT-compatible features.
  • The pure single-stream DiT jointly processes image and condition tokens through shared Transformer blocks and self-attention.This allows semantic conditions and evolving visual tokens to interact at every layer.
  • Editing combines SigLIP-VQ semantic features with clean VAE latents to improve instruction following and reference consistency.The semantic pathway supplies high-level guidance, while the pixel pathway preserves identity, structure, texture, and background.

3 Data Collection and Preparation

The data pipeline emphasizes large-scale real images, using image-only data for visual-prior learning and paired image–text data for supervised fine-tuning. Filtering and consistency checks prepare image, caption, and editing pairs for training.

  • Real image-only data supports visual-prior learning at scale without requiring paired captions.The recipe uses image-only data for pre-training and mid-training, while paired image–text data is reserved for SFT.
  • 98% of 220M generation-training samples are real images, and image-only samples exceed 90% of the total.Pre-training and mid-training use only real images; real images remain above 70% of paired SFT data.
  • The SFT corpus is organized into nature, design, people, and synthetic-content groups.Figure 4 summarizes the SFT-weighted composition across these groups.
  • Image filtering applies metadata, aesthetic, and quality criteria before training crops and resolutions are assigned.The passage specifies pixel-count, file-size, ArtiMuse, and DeQA-Score thresholds.
  • Automatically generated captions are retained only when image quality and caption agreement are sufficient.The checks remove hallucinated content, inaccurate text transcriptions, malformed outputs, refusals, repetitions, private information, and watermarks.
  • Editing pairs are kept only when instructions describe visible source-to-target changes correctly and contain no unsupported details.

4 Training Recipe

The training recipe strengthens the understanding backbone before generation training and uses Muon across generation stages. Detailed configuration tables document the independent understanding and generation pipelines.

  • The training path begins with Chain-of-Thought supervised fine-tuning to enhance the understanding backbone before generation training.The overall release path then proceeds through the documented generation and understanding pipelines.
  • Muon is used across all generation stages.
  • Table 1 provides the detailed configuration for understanding training and identifies CoT SFT as CoT Supervised Fine-tuning.

4.1 CoT Supervised Fine-tuning

Understanding training packs multiple data types into long sequences and applies block-diffusion masking to supervised response regions. AdamW with sharded mixed-precision training and checkpointing supports optimization.

  • Generation, understanding, and text-only data are mixed at a 9:9:2 ratio in sequences of exactly L = 16,384 tokens.Offline tokenization, greedy bin-packing, and block-diagonal attention prevent cross-sample leakage.
  • The block-diffusion objective masks response tokens while leaving system prompts, questions, and visual tokens unmasked and unsupervised.Loss is computed over masked assistant-turn positions, including the <think> trace and answer.
  • Each batch is consumed twice using complementary masks so every response token is supervised across the pair.The loss is normalized per block before averaging across supervised blocks and packed sequences.
  • AdamW training uses FSDP2 full sharding, mixed precision, and gradient checkpointing with linear warmup and cosine decay.Because complementary passes take separate optimizer steps, the schedule spans twice the number of data iterations.

4.2 Image-only Pre-training

Image-only pre-training bootstraps the visual generator from unlabeled images using masked self-conditioning and flow matching, avoiding large-scale paired image–text supervision.

  • Image-only supervision: Large-scale image–text pairs are not required to pre-train LLaDA-Image; images alone supervise a strong visual generative prior.A frozen vision-language model extracts semantic self-conditioning signals from the training images.
  • Resolution-aware sampling: High-resolution images are randomly cropped before resizing to 256^2, preserving local detail while keeping the condition aligned with the training target.The VLM derives the semantic condition from the processed crop rather than an aggressively downsampled full image.
  • Masked self-conditioning: Masked self-conditioning hides image tokens so the generator must recover missing content from visible patches and learn contextual visual structure.The image features are paired with an auxiliary prompt, then masked with keep probability 1 − ρ_img.
  • Optimization: Flow matching trains the generator on interpolations between clean latents and Gaussian noise, while only the generator and conditioning adapters are updated.The vision encoder and vision-language model remain frozen throughout this stage.

4.3 Image-only Mid-training

Image-only mid-training bridges low-resolution visual-prior learning and later text alignment by preserving image-derived conditioning while increasing spatial detail and resolution flexibility.

  • Resolution transition: 512^2 mid-training retains image-derived self-conditioning, token masking, and flow matching while increasing the nominal pixel budget from 256^2 to 512^2.The target pixel budget is not restricted to a fixed square shape.
  • Alignment bridge: This stage bridges image-only pre-training and paired text-to-image alignment without introducing caption mismatch during the resolution transition.Captions describing complete images can lose fine-grained details when the images are heavily downsampled.
  • Variable-resolution training: Aspect-ratio buckets assign images to comparable pixel budgets while preserving source geometry and exposing the model to diverse spatial shapes.The strategy also keeps memory and computation closely matched across data-parallel ranks.

4.4 Supervised Fine-tuning

Supervised fine-tuning progressively aligns text with the visual prior, raises resolution, targets text and portrait capabilities, and jointly trains generation with editing under a real-data-dominant mixture.

  • Optimization: Logit-normal timestep sampling shifts supervision toward larger t, concentrating training on the more challenging high-noise regime.The reported settings are P_mean = 0.8 and P_std = 0.8, while retaining support across the denoising trajectory.
  • Data composition: A real-image share above 70% throughout supervised fine-tuning trades slower early convergence for substantially stronger realism after sufficient training.The authors report that the advantage becomes increasingly apparent over longer training.
  • Progressive alignment: Text-image alignment begins at approximately 512^2 pixels and then transitions to approximately 1024^2 pixels for finer textures and sharper boundaries.Aspect-ratio buckets preserve varied image shapes at both pixel budgets.
  • Targeted refinement: Targeted refinement increases text-intensive and portrait-centric examples to strengthen text rendering and high-fidelity human generation.Synthetic data is deliberately limited throughout the supervised fine-tuning stages.
  • Unified generation and editing: Joint editing training mixes text-to-image and image-to-image examples at a 1:1 ratio to avoid overfitting editing and weakening open-ended generation.Reference images enter through a parallel SigLIP-VQ-based stream rather than the vision-language pathway.
  • Checkpoint merging: Checkpoint merging averages nearby converged states to suppress transient parameter deviations and stabilize performance across evaluation benchmarks.The procedure preserves shared capabilities learned across neighboring checkpoints.

4.5 Few-step Distillation

TwinFlow distills the multi-step LLaDA-Image into a few-step generator by alternating fake-score estimation and distribution-matching updates within a shared DiT.

  • Distillation framework: TwinFlow distills the original multi-step model into LLaDA-Image Turbo using distribution-matching distillation with frozen real-score and online fake-score models.The fake-score estimator tracks the non-stationary student distribution and uses the faster optimization timescale.
  • Student mapping: The student starts from Gaussian noise and uses a direct one-step mapping to a clean sample under the flow-matching convention.The predicted velocity is interpreted as z − x for the noise-to-data update.
  • Fake-score training: Re-noised detached student samples train the negative-time fake-score branch, with gradients restricted to its prediction.The stop-gradient operation prevents this objective from updating the generated sample pathway.
  • Distribution matching: The distribution-matching objective re-noises differentiable student samples and compares converted velocity predictions from frozen and student branches.Alternating these updates trains the positive-time generator and negative-time fake-score branch without a separate auxiliary score network.
  • Shared architecture: Two output heads share one DiT backbone: the negative-time head estimates fake scores, while the positive-time head parameterizes the distilled generator.The time-sign convention indicates learning roles rather than separate time domains.

5 Experiments

LLaDA-Image is evaluated across general generation, bilingual text rendering, and instruction-based editing using standard inference settings and multiple benchmarks. Results are strongest on Qwen-Image-Bench, while text rendering and editing reveal both broad capability and specific remaining gaps.

  • Evaluation Protocol: All LLaDA-Image Turbo benchmark results use four sampling steps, excluding prompt enhancement, explicit reasoning, and test-time prompt rewriting.This evaluation protocol is intended to compare models under standard inference settings.
  • General Image Generation Performance: 53.53 and 53.38 overall scores on Qwen-Image-Bench English and Chinese tracks exceed Z-Image Turbo by 1.87 and 0.67 points, respectively.LLaDA-Image establishes a new open-source state of the art on both tracks without prompt enhancement or test-time thinking.
  • Text Rendering: 0.923 and 0.913 LongText-Bench scores on English and Chinese subsets indicate balanced bilingual long-text rendering.The scores remain below the strongest dedicated text-rendering models, so LongText-Bench is paired with CVTG-2K for broader evaluation.
  • Text Rendering: 0.875 average word accuracy, 0.945 NED, and 0.818 CLIPScore on CVTG-2K show strong multi-region visual-text generation.Word accuracy decreases from 0.892 with two text regions to 0.857 with five, while remaining relatively stable as region count increases.
  • Legacy Diagnostic Benchmarks: 0.85 overall GenEval reflects strong single-object generation and two-object composition, but the 0.53 Counting score exposes a compositional weakness.GenEval is reported as a traditional fine-grained reference rather than treated as faithful evidence of general generation quality.
  • Instruction Following and Editing: 87.48 on DPG-Bench and GEdit-Bench scores of 7.336 English and 7.294 Chinese demonstrate strong dense-prompt following and bilingual editing.On GEdit-Bench, semantic consistency is more competitive than perceptual quality, particularly in English, where scores are 8.043 versus 7.182.

6 Conclusion and Future Directions

LLaDA-Image combines an open, inspectable training recipe with unified generation and instruction-guided editing. Remaining challenges concern world knowledge, complex text rendering, and native 2K generation.

  • Conclusion: LLaDA-Image is a 6B Diffusion Transformer trained from scratch for unified text-to-image generation and instruction-guided editing.The released recipe includes image-only pre-training and mid-training, a dLLM-based VLM, progressive training, and TwinFlow distillation.
  • Conclusion: 220M generation samples comprise 98% real images, while more than 90% support image-only training and real images remain above 70% in paired SFT.The pipeline also uses SigLIP-VQ, clean VAE latents, parameter-free RMSNorm, and Muon.
  • Conclusion: LLaDA-Image achieves state-of-the-art results among open-source models on both Qwen-Image-Bench tracks and supports 2–4-step Turbo inference.The conclusion also reports strong bilingual text rendering and editing.
  • Future Directions: Remaining challenges include broader world knowledge and reliable long or multi-region text rendering.Text stability decreases with more regions, longer requested content, and more complex layouts.
  • Future Directions: Native 2K generation remains an extension rather than an evaluated capability because training currently reaches a nominal 1024^2 pixel budget.Future work targets native 2K training, stronger grounding and typography, broader knowledge coverage, and agentic generation.

A Aspect-Ratio Bucket Configurations

Aspect-ratio buckets preserve image geometry while organizing training around approximate pixel budgets. Images are assigned to the closest bucket, resized without changing aspect ratio, and cropped only for small overflow.

  • Aspect-Ratio Bucket Configurations: 512^2-pixel and 1024^2-pixel configurations report approximate image areas rather than fixed square training resolutions.Each configuration is represented as a (W, H) tuple in pixels.
  • Bucket Assignment: For an image with aspect ratio a = W/H, the loader assigns the image to the bucket with the smallest absolute aspect-ratio difference.The bucket set is B = {(W_k, H_k)}_K.
  • Resizing and Cropping: The selected bucket determines a resize-to-fill operation that preserves the source aspect ratio before producing the target shape.One resized dimension matches the bucket, while the other may exceed it slightly.
  • Resizing and Cropping: Cropping removes only the small overflow after resizing, typically deleting a few pixels and not materially altering the image.The close aspect-ratio match keeps overflow limited.
  • Systems Motivation: Approximately equal bucket areas let data-parallel ranks process different aspect ratios with similar image-token counts.This reduces memory peaks and training-time stragglers while avoiding square-induced distortion and aggressive cropping.

B CoT Supervised Fine-tuning and Reinforcement Learning

The CoT supervised fine-tuning stage is isolated because it is the only stage that updates the understanding backbone. Its evaluation spans 18 multimodal benchmarks covering general VQA, reasoning, OCR/document understanding, and other multimodal tasks.

  • CoT Supervised Fine-tuning: CoT supervised fine-tuning is the only stage that updates the understanding backbone, so its effect is verified separately before generation training.Later generation stages build on this separately assessed component.
  • Evaluation: 18 multimodal benchmarks evaluate general VQA, reasoning, OCR/document understanding, and other multimodal capabilities.The benchmark suite includes general, reasoning, OCR/chart, and additional multimodal task groups.
  • Evaluation: General-task evaluation includes MMStar, MMBench, HallusionBench, RealWorldQA, and SimpleVQA.MMBench uses English and Chinese development splits.
  • Evaluation: Reasoning-task evaluation includes MMMU, MathVista, We-Math, and MathVision.The cited configurations include validation or mini splits where specified.
  • Evaluation: OCR and chart evaluation includes ChartQA, DocVQA, InfoVQA, OCRBench, and AI2D.These tasks target chart, document, information, OCR, and diagram understanding.
  • Evaluation: Additional multimodal evaluation includes CountBenchQA, VL-RewardBench, and V∗.These benchmarks form the other multimodal task group.
  • Results: Table 11 compares multimodal understanding results before and after CoT supervised fine-tuning.The LLaDA 2.0 Uni column reproduces technical-report scores, while CoT SFT is evaluated under the report’s protocol.

C Additional Qualitative Comparison

The qualitative comparison presents generated outputs for historical artifacts, travel scenes, towns, and landscapes. These examples span Chinese and English descriptions and illustrate the report’s visual-generation evaluation material.

  • Additional Qualitative Comparison: A generated output depicts a Han-dynasty pottery female attendant with archaeological surface details under soft museum lighting.The prompt specifies faded pigments, cracks, soil staining, damage, and a vertical composition.
  • Additional Qualitative Comparison: A generated output depicts a Ming-dynasty cloisonné begonia-shaped double-handled vase with gilded dragon handles and dense multicolored decoration.The description emphasizes ornate form, blue-green ground, and fine gold-wire patterns.
  • Additional Qualitative Comparison: A generated travel photograph shows a male visitor on a glacier viewpoint with snow mountains, exposed ice, rock, and natural daylight.The scene combines a detailed subject description with a broad glacial landscape.
  • Additional Qualitative Comparison: A realistic European mountain-town photograph depicts layered stone houses, winding streets, terraces, distant mountains, and water in morning light.The composition emphasizes natural colors and a lived-in travel-photography appearance.
  • Additional Qualitative Comparison: A realistic Jiangnan water-town scene centers on a winding canal, traditional residences, a stone arch bridge, a wooden boat, and a teahouse sign.The prompt includes layered streets, shops, vegetation, mist, and everyday life.
  • Additional Qualitative Comparison: A wide-angle coastal-cliff photograph uses a high viewpoint to show grassland, eroded cliffs, deep-blue sea, wave foam, and distant headlands.The scene contains no people or buildings and emphasizes scale and open space.

D Image Captioning and Filtering

The section describes bilingual captioning prompts and filtering rules for constructing faithful text-to-image training annotations. It combines structured descriptions, verbatim text transcription, and quality checks for privacy, watermarks, and caption faithfulness.

  • Filtering: Filtering removes failed outputs, watermarks, private information, inaccurate text transcriptions, and captions that hallucinate visual content.A unified review assigns privacy and watermark booleans plus transcription and caption-faithfulness categories.
  • Captioning: Chinese annotations must preserve visible text verbatim, retain its original language, and quote it in detailed prompts.The Chinese instructions also require exact transcription in the render text field.
  • Captioning: Short prompts focus on the most important image elements and simulate concise natural user inputs.The Chinese guidance targets roughly 10–50 characters, while the English guidance targets roughly 5–30 words.
  • Captioning: Captioning prompts require faithful, specific, information-dense descriptions grounded only in visible image content.The prompts cover subjects, scenes, composition, style, lighting, color, materials, and visible text.
  • Captioning: English annotations similarly require direct visual descriptions, verbatim text transcription, and JSON-only output.Descriptions should vary their opening, avoid invented details, and scale with image complexity.

E Prompts Used in The Report

The report presents detailed prompts for portraits, couples, markets, landscapes, and poster-like compositions. These examples specify subjects, settings, composition, lighting, materials, colors, and visible text to guide image generation.

  • Portrait prompts: The prompt set includes naturalistic portraits in parks, cafés, libraries, restaurants, and other everyday settings.Examples specify appearance, clothing, pose, facial expression, environmental details, and lighting conditions.
  • Scene prompts: A market prompt describes a middle-aged stallholder arranging colorful vegetables in early-morning light.It includes produce details, another customer, market structures, handwritten pricing text, and realistic tonal depth.
  • Scene prompts: A seaside travel prompt depicts two young men and one young woman posing on a wooden boardwalk at evening.The prompt specifies clothing, facial features, wind, ocean scenery, a lighthouse, rocks, shrubs, and soft overcast light.
  • Scene prompts: A restaurant example depicts a young East Asian couple celebrating a birthday with a pink cake and warm interior details.The scene includes candles, food, tableware, wine, flowers, plants, wood paneling, and soft warm lighting.
  • Poster prompt: The poster prompt controls negative space, a vintage Eastern palette, concentrated golden-yellow moonlight, and print-like textures.It specifies approximate hex colors, screen-printing and woodblock effects, registration misalignment, halftone grain, and uneven ink density.
Loading 2609.03796v1…