PaperlayerView exact revision on arXiv

Source-linked AI summary

Know, then Draw: A New Learning Paradigm for Visual Generation

Yucheng Mao, Chongjian Ge, Zhuowen Tu, Hao Tan, Hanwen Jiang

arXiv:2610.37626v1cs.CV

TL;DR

Visual generation requires both high-level semantic knowledge and detailed image rendering, but autoregression and diffusion are often compared as competing paradigms. This paper assigns knowledge acquisition to autoregressive pretraining and rendering to diffusion post-training, reporting stronger generators across evaluated protocols and a compute-efficient text-to-image result.

  • Problem

    Visual generation must acquire high-level visual knowledge and render it into detailed images, but autoregressive and diffusion paradigms are typically compared as competitors rather than as solutions to these distinct tasks.

    Source: §1 · pp. 1–2

  • Method

    The framework uses autoregressive pretraining to acquire visual knowledge and diffusion post-training to render images, with scaling laws guiding compute allocation between the stages.

    Source: §1 · p. 2

  • Results

    The specialization yields stronger generators across evaluated protocols and compute budgets; the model achieves 0.86 on GenEval and 87.78 on DPG-Bench with approximately 6,600 H100 GPU-hours.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Semantic quality matters more than reconstruction or AR-only generation quality for post-training performance, motivating language-supervised semantic tokens without reconstruction loss.

    Source: §1 · p. 2

    The pipeline comparison includes both pretrained weights and conditioning tokens extracted from target images, so it does not isolate their individual contributions.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

Visual generation requires both high-level semantic knowledge and fine-grained pixel details. We propose a learning paradigm that assigns these roles to autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Within this framework, we study the objective, representation, and scaling of knowledge acquisition and rendering. Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality. We further find that AR-only generation quality does not necessarily correlate with final performance; semantic quality matters more. This motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss. We establish compute-optimal scaling laws for knowledge acquisition and rendering, and find that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch. Guided by the laws, we train a 3B-parameter text-to-image model with only 6,600 H100 GPU-hours, achieving 86 on GenEval and 87 on DPG-Bench and validating the proposed paradigm at scale.

1 INTRODUCTION

Visual generation needs both high-level visual knowledge and detailed rendering, although autoregressive and diffusion models are often compared as competing approaches. The paper argues that these tasks favor different objectives and reports results supporting a specialized, compute-guided pipeline.

1 INTRODUCTION

  • Autoregressive pretraining acquires visual knowledge, while diffusion post-training renders images; this specialization yields stronger generators across evaluated protocols and compute budgets.At inference, autoregressive tokens provide a structured visual representation that conditions diffusion.Source: §1 · p. 2
  • Semantic quality, rather than reconstruction quality or AR-only generation quality, determines post-training generation performance across the evaluated tokenizers.This motivates Semantic-LM, trained with language supervision without an image decoder or reconstruction loss.Source: §1 · p. 2
  • Scaling laws guide compute allocation between pretraining and post-training, and diffusion post-training has lower fitted asymptotic rendering loss than pixel diffusion trained from scratch at matched model and data scales.The reported lower asymptotic loss suggests benefits beyond faster convergence.Source: §1 · p. 2
  • 0.86 on GenEval and 87.78 on DPG-Bench were achieved by a roughly 3B-parameter text-to-image model trained with approximately 6,600 H100 GPU-hours.Its DPG-Bench performance was comparable to Z-Image-Base using approximately 1/48 of that model’s estimated training compute.Source: §1 · p. 2

2 METHOD

The method separates visual knowledge acquisition from pixel rendering: an autoregressive model learns discrete visual tokens, then a diffusion renderer reuses its frozen backbone. The paper also compares tokenizer targets and uses scaling laws to guide stage-specific compute allocation.

2 METHOD

  • A frozen visual tokenizer converts images into discrete tokens for text-conditioned autoregressive prediction; the backbone is then frozen and augmented with pixel-diffusion modules that reuse its attention and representations.At inference, autoregressively generated tokens from the text prompt condition the renderer.Source: §2 · p. 2

2.1 AUTOREGRESSIVE PRETRAINING FOR KNOWLEDGE ACQUISITION

  • The tokenizer determines autoregressive prediction targets, and the experiments compare reconstruction-based and semantic tokenizers, including Semantic-LM’s language-modeling supervision without an image decoder or reconstruction loss.The comparisons test semantic representations and whether reconstruction supervision is necessary for effective semantic tokens; tokenizers remain frozen during generator training.Source: §2.1 · p. 3, Figure 3 · p. 3

2.2 PIXEL-DIFFUSION POST-TRAINING FOR RENDERING

  • For rendering, trainable pixel-diffusion modules reuse frozen backbone attention projections, while the renderer is trained with pixel-space flow matching and velocity-space mean squared error.Post-training conditions on tokens extracted from the ground-truth image; inference instead uses tokens generated autoregressively from the text prompt.Source: §2.2 · pp. 3–4

2.3 SCALING METHODOLOGY

  • Objective-specific scaling laws fit autoregressive cross-entropy and diffusion velocity-MSE using C = 6ND, then translate compute-optimal stage settings into an allocation between pretraining and post-training.The diffusion compute proxy excludes the frozen autoregressive forward pass; training recipes are tuned to transfer across model sizes within each family.Source: §2.3 · p. 4

3 EXPERIMENTS

The experiments compare pretraining objectives, tokenizer targets, and compute scaling, then test a scaling-guided recipe at larger scale. They favor autoregressive knowledge acquisition, semantic targets, and diffusion rendering, with competitive text-to-image scores at about 6,600 H100 GPU-hours.

3.1 AUTOREGRESSION OR DIFFUSION FOR VISUAL KNOWLEDGE ACQUISITION?

  • After 100k post-training updates, AR→PDM reaches 64.8 on GenEval versus 61.2 for PDM trained for 400k iterations; further post-training reaches 65.5 on GenEval and 71.1 on DPG-Bench.Direct LDM reaches 57.3 on GenEval, while LDM→PDM achieves 56.6, indicating that two-stage training alone is insufficient under the evaluated settings.Source: Table 1 · p. 5, §3.1 · p. 5

3.2 WHAT SHOULD AUTOREGRESSIVE PRETRAINING PREDICT?

  • Recon’s better reconstruction yields lower final scores than Semantic-Recon: 54.7 versus 65.5 on GenEval and 63.0 versus 71.1 on DPG-Bench.Semantic-LM, without reconstruction supervision, reaches 64.8 on GenEval and 67.7 on DPG-Bench after post-training.Source: Table 2 · p. 5, §3.2 · p. 6
  • Recon’s AR-only GenEval score of 57.49 exceeds Semantic-Recon’s 50.44, but after diffusion post-training the ranking reverses: 54.67 versus 65.51.The reversal also appears on DPG-Bench; Semantic-LM has no decoder yet supports effective diffusion rendering.Source: Figure 4 · p. 6, §3.2 · p. 6

3.3 SCALING KNOWLEDGE ACQUISITION AND RENDERING

  • Pretraining model size and data scale at approximately matched rates, with Nopt ∝ C^0.47–0.53 and Dopt ∝ C^0.47–0.53.Rendering estimates vary more across methods: Nopt ∝ C^0.45–0.70 and Dopt ∝ C^0.30–0.55; the rendering law is for paired renderers and frozen AR backbones.Source: Figure 5 · p. 7, §3.3 · p. 7
  • Fitted asymptotic velocity-MSE is 0.070 for diffusion post-training versus 0.0788 for scratch pixel diffusion; the comparison evaluates the full pipeline, not isolated contributions.The asymptotic terms are extrapolated trends under the fitted scaling models.Source: §3.3 · pp. 7–8

3.4 VALIDATING THE LEARNING PARADIGM AT SCALE

  • At about 6,600 H100 GPU-hours, the roughly 3B-parameter model scores 0.86 on GenEval and 87.78 on DPG-Bench, with comparisons approximate across differing systems and settings.Only Qwen-Image and Seedream 3.0 report higher DPG-Bench scores among the listed baselines; the results do not establish that the compute allocation is optimal.Source: Table 3 · p. 8, §3.4 · pp. 8–9

3.3 SCALING KNOWLEDGE ACQUISITION AND RENDERING

  • The induced compute allocation is nearly scale-invariant: Cpre : Cpost ≈ 3:2, or about 60% for knowledge acquisition and 40% for rendering.The figure plots compute-optimal post-training share against model size.Source: Figure 6 · p. 8, §3.3 · p. 8

4 RELATED WORK

Prior visual-generation work generally treats autoregression and diffusion as competing paradigms, while this paper assigns them distinct roles in knowledge acquisition and rendering. It also distinguishes its objective separation from prior combinations and studies compute allocation across objectives.

Diffusion-based visual generation

  • Earlier work enriched diffusion representations with semantics, but kept semantic learning coupled to reconstruction or diffusion; this paper instead trains Semantic-LM using language supervision alone.Semantic-LM uses no image decoder or reconstruction loss.Source: §4 · p. 9

Autoregressive visual generation

  • Autoregressive image-generation methods predict learned visual tokens, but their representations must support synthesis; this work frees them from reconstruction and AR-only generation demands.Prior methods include raster, next-scale, and randomized prediction [Sun et al., 2024; Tian et al., 2024; Pang et al., 2024].Source: §4 · p. 9

Combining autoregression and diffusion

  • The paper separates autoregressive visual-knowledge acquisition from diffusion rendering, unlike prior approaches that keep autoregression involved in image synthesis or couple the objectives differently.Related approaches include MAR [Li et al., 2024], Transfusion [Zhou et al., 2024], and concurrent VoT [Sun et al., 2026].Source: §4 · p. 9

Scaling laws for generative models

5 CONCLUSION

The conclusion presents a visual-generation pipeline that separates semantic knowledge acquisition from pixel rendering. Its scaling analysis and text-to-image results support this stage-specific approach.

5 CONCLUSION

  • Semantic tokens provide effective pretraining targets without image reconstruction or standalone AR generation.The conclusion also reports that stage-specific scaling laws guide compute allocation and that the fitted asymptotic rendering loss is lower for the pretrained pipeline.Source: §5 · pp. 9–10
  • The proposed paradigm assigns visual-knowledge learning to autoregressive pretraining and pixel rendering to diffusion.The conclusion frames the approach as tailoring the objective, representation, and compute budget to each stage.Source: §5 · p. 10
  • 0.86 on GenEval and 87.78 on DPG-Bench were achieved with approximately 6,600 H100 GPU-hours of training.The conclusion presents this as a compute-efficient text-to-image generator.Source: §5 · p. 10

A ADDITIONAL EXPERIMENTAL DETAILS

This appendix section states that it provides experimental training and evaluation details.

A ADDITIONAL EXPERIMENTAL DETAILS

A.1 SEMANTIC TOKENIZER TRAINING

The appendix describes two semantic tokenizers that share a frozen visual encoder and IBQ codebook approach but receive different training signals. Semantic-Recon uses reconstruction objectives, whereas Semantic-LM uses language supervision without image reconstruction.

A.1 SEMANTIC TOKENIZER TRAINING

  • Both tokenizers quantize post-merger features from a frozen Qwen2.5-VL-32B-Instruct visual encoder using an IBQ codebook, and remain frozen during generator training.Tokenizer training is separate from the reported AR-pretraining and diffusion-post-training budgets.Source: §A.1 · p. 15
  • Semantic-Recon trains an IBQ codebook and decoder to reconstruct images using reconstruction, VGG-based LPIPS perceptual, quantization, and entropy-regularization losses.The decoder has 12 layers, width 768, and 12 attention heads; training runs for 80,000 updates.Source: §A.1 · p. 15
  • Semantic-LM trains only the IBQ codebook, using next-token cross-entropy from supervised text targets and entropy regularization, with no image decoder or reconstruction loss.The frozen language model is conditioned on quantized visual features; training uses 200B DataComp-VLM for 40,000 updates.Source: §A.1 · p. 15

A.2 CONTROLLED GENERATION COMPARISONS

The controlled generation comparisons match model size, image exposure, and training samples while varying the route or tokenizer. Pretraining–post-training models use autoregressive pretraining followed by diffusion post-training, with stage durations and resolution curriculum specified.

A.2 CONTROLLED GENERATION COMPARISONS

  • Pretraining–post-training models receive 150,000 autoregressive updates followed by 250,000 diffusion post-training updates.Direct-pixel and AR controls each receive 400,000 optimizer updates.Source: §A.2 · p. 15
  • The comparison routes use approximately 400M final parameters and match cumulative image exposure; pretraining–post-training budgets include both stages.The AR tokenizer variants share the AR→PDM recipe.Source: Table 4 · p. 16
  • The first 50,000 updates use the low-resolution tier, while the remaining 350,000 mix low- and high-resolution tiers.Thus pretraining comprises 50,000 low-resolution and 100,000 mixed-resolution updates; all post-training updates use the mixed-resolution phase.Source: §A.2 · p. 16
  • Controlled models see the same samples in the same order, with repeated presentations counted across both stages.The shared diffusion objective uses the same noising, resolution shift, and loss convention for PDM and diffusion post-training.Source: §A.2 · p. 17
  • Tokenizer comparisons hold the generator and training recipe fixed, freeze each tokenizer, and vary its visual representation and codebook supervision.Semantic-LM has no image decoder, so its reconstruction quality is undefined; AR-only generation is evaluated after 150,000 pretraining updates.Source: §A.2 · p. 17

A.3 PRETRAINING HYPERPARAMETER TRANSFER

The pretraining transfer study tests learning rates across width and depth under fixed architecture-scaling and initialization rules. A 7 × 10^-5 base rate is near-optimal across the tested architectures.

A.3 PRETRAINING HYPERPARAMETER TRANSFER

  • The sweep varies width at 16 layers and depth at width 512 across 11 causal RandAR transformer architectures.The models share frozen FLAN-T5-XL text features and a frozen Semantic-Recon tokenizer; only the AR generator is optimized.Source: §A.3 · pp. 17–18
  • Each architecture is tested at 12 base learning rates, from 10^-5 to 3 × 10^-3, under fixed CompleteP-style initialization and parameter-scaling rules.The experiment varies the base learning rate, not the initialization scale.Source: §A.3 · p. 18
  • 7 × 10^-5 gives the grid-minimum training cross-entropy at all six depths, with maximum excess of 0.0048 nats per token across 11 architectures.The result uses the 50,000-update sweep and averages 21 raw training cross-entropy measurements per plotted point.Source: §A.3 · p. 19, Figure 7 · p. 19

A.4 AR SCALING PROTOCOL AND RESULTS

The AR scaling study fits loss and compute-allocation relationships across model sizes using training-curve envelopes, IsoFLOP profiles, and a parametric loss model. Its fitted coefficients define the full-sequence AR allocation laws.

A.4 AR SCALING PROTOCOL AND RESULTS

  • The envelope method interpolates loss in log compute, chooses the minimum over 1,600 budgets, and fits power laws to the winning model and data sizes.The envelope spans 1.5535 × 10^18–2 × 10^21 FLOPs; IsoFLOP uses 12 budgets over 5 × 10^18–3 × 10^20 FLOPs.Source: §A.4 · pp. 19–20
  • The parametric fit uses 432 equally weighted loss samples and optimizes a Huber objective across 75 initializations with positive coefficients and exponents.The optimal allocation exponents are a = β/(α + β) and b = 1 − a.Source: §A.4 · p. 20
  • E = 5.7238, A = 0.3194, B = 1.7284, α = 0.4707, and β = 0.4386 are the fitted parametric loss coefficients.The parametric model is bL = E + An^-α + Bd^-β, with normalized model size n and data d.Source: §A.4 · p. 20
  • The full-sequence AR allocation laws relate optimal model size N and token count D to compute C as N = N20(C/10^20)^a and D = D20(C/10^20)^b.Tab. 7 gives the full-sequence allocation laws.Source: Table 7 · p. 20, §A.4 · p. 20

A.5 POST-TRAINING SCALING PROTOCOL AND RESULTS

The post-training study scales diffusion renderers attached to frozen autoregressive backbones and fits laws for renderer parameters and training data. The shared 8 × 10^-4 learning rate is close to each tested model's observed minimum.

A.5 POST-TRAINING SCALING PROTOCOL AND RESULTS

  • The scaling family uses native images at the 256^2 area tier, 14-pixel diffusion patches, and frozen pretrained AR backbones; training uses base learning rate 8 × 10^-4 without warmup or learning-rate decay.Only the diffusion FFNs, AdaLN modules, and input/output adapters are newly initialized and trained.Source: §A.5 · pp. 20–21
  • Observed learning-rate minima, in increasing model-size order, are 10^-3, 6 × 10^-4, 6 × 10^-4, and 8 × 10^-4; the shared scaling rate 8 × 10^-4 is within 0.0003 velocity-MSE of each model’s minimum.Figure 8 compares mean training loss at approximately 20,000 updates for four width/depth pairs; each point averages 41 measurements, and the largest model uses its final available windows.Source: §A.5 · p. 21, Figure 8 · p. 21
  • The renderer law counts trainable diffusion parameters but excludes the frozen AR path and external encoders; each image has four diffusion patches per AR code.The sequence count is Dseq = Dtext + 2DAR + Dtgt = Dtext + 1.5Dtgt.Source: §A.5 · p. 21
  • The fitted post-training scaling law is conditional on the source AR checkpoint family.The source checkpoint varies across models.Source: §A.5 · p. 22

A.6 PIXEL-DIFFUSION HYPERPARAMETER TRANSFER

The appendix reports pixel-diffusion scaling and compute-allocation analyses, then describes the final model’s training budget and discusses semantic inference and rendering. The analyses identify a stage-budget split and compare the fitted asymptotic loss of pixel diffusion trained from scratch with post-training.

A.6 PIXEL-DIFFUSION HYPERPARAMETER TRANSFER

  • Across the original pixel-diffusion width and depth searches, every curve reached its grid minimum at either 8 × 10−4 or 10−3.The search statistic was mean training velocity-MSE over the final 10,000 updates; additional anchor-architecture checks also had minima at those rates.Source: §A.6 · p. 23, Figure 9 · p. 23

A.7 PIXEL-DIFFUSION SCALING FROM SCRATCH

  • The fitted asymptotic loss for pixel diffusion trained from scratch exceeds the post-training value Epost = 0.0701.The appendix reports envelope, IsoFLOP, and parametric allocation analyses for the from-scratch pixel-diffusion scaling study.Source: §A.7 · p. 24, Figure 10 · p. 25

A.8 COMPUTE ACCOUNTING AND STAGE ALLOCATION

  • The induced compute-optimal split across models with at least 261M parameters was Cpre : Cpost = 1.50:1 (r = 0.40), using a normalized 6ND proxy.The proxy excludes the frozen-AR forward pass during post-training, and smaller models were omitted because their splits were unstable.Source: §A.8 · p. 25, Table 13 · p. 25

A.9 FINAL MODEL RECIPE AND EVALUATION

  • Approximately 6,600 H100 GPU-hours were allocated to final-model training, with a prescribed 3:2 pretraining-to-post-training budget ratio.The nominal stage budgets were approximately 3,960 and 2,640 GPU-hours, respectively; these are allocations, not measured runtimes.Source: §A.9 · p. 26

A.10 DISCUSSION: SEMANTIC INFERENCE AND RENDERING IN FRONTIER MODELS.

  • The discussion frames autoregression as resolving what should be rendered, while diffusion realizes that specification in pixels; current experiments do not determine which visual-token interface is preferable with rich language or agentic context.The authors distinguish Semantic-LM’s language-aligned representation from Semantic-Recon’s reconstruction-oriented visual information and leave their interaction with rich context for future work.Source: §A.10 · p. 26

Related papers

Loading 2610.37626v1…