Intrinsic Distribution Matching

1 Introduction

• Long-text rendering tests whether one- or few-step generation preserves detailed conditional information, including legible and correct characters in a coherent image.The paper evaluates full-parameter and LoRA tuning on Z-Image-Turbo 6B and Qwen-Image 20B.Source: §1 · pp. 1–2
• The method retains the full DMD correction while adding explicit moment supervision, with ablations testing its benefit in a shared training setting.The feature-moment constraint is motivated by a decomposition that separates moment-determined score structure from a residual.Source: Figure 1 · p. 2, §1 · p. 2
• IDM augments internal DMD with matching the means and per-coordinate standard deviations of generator-EMA features.The EMA representation is shared by generated and reference samples, avoiding an external feature encoder.Source: §1 · p. 2
• 94.25% on LongText-CN and 90.20% on LongText-EN are the highest scores among compared methods at 2 NFE.These improve over Z-Image-Turbo++ by 2.63 and 0.32 percentage points, respectively.Source: §1 · p. 2

Rectified Flow and Few-Step Generation

• At one network evaluation, the rectified-flow generator maps noise and a text condition to an output as Gθ(z, c) = z − Fθ(z, 1, c).The velocity network regresses the pair velocity ϵ − x_r; few-step generation integrates the field from noise to data with forward-Euler steps.Source: §2 · p. 4

Distribution Matching Distillation

• DMD uses the difference between reference and generated-distribution scores to guide generated samples toward the reference distribution.A pretrained teacher supplies the reference score, while a fake-score network estimates the generated-distribution score.Source: §2 · p. 4

Internal Score Estimation

• TwinFlow and APEX estimate the reference and model scores with internally separated velocity branches, removing the separate fake-score network while retaining model-score learning.The paper follows this internal construction for DMD.Source: §2 · p. 4

3.1 Reverse-KL Field

• The reverse-KL field is the model score minus the reference score, and its negative gives the sample-space descent direction toward a lower model-to-reference density ratio.The corresponding noisy field is the score difference estimated in DMD.Source: §3.1 · p. 4

3.3 Decomposition of the Reverse-KL Field

• The reverse-KL field decomposes into a Gaussian component determined by means and covariances and a residual that can remain nonzero when those moments match.The decomposition also applies to the noisy field used in DMD.Source: Figure 3 · p. 5, §3.3 · p. 5

3.2 Parameter Gradient

• Backpropagating the sample-update field through the generator maps the distributional direction into a parameter update.For reverse KL, the first-variation potential recovers the score-difference field.Source: §3.2 · p. 5

3.3 Decomposition of the Reverse-KL Field

• Because moment matching alone cannot capture the residual, training retains full DMD correction and adds a loss on generator-EMA feature means and standard deviations.The added feature-space objective is motivated by the decomposition, not a separate estimate of its sample-space Gaussian or residual component.Source: §3.3 · p. 6

4 Intrinsic Distribution Matching

• IDM retains the full internal DMD correction and adds a loss matching generator-EMA feature means and per-coordinate standard deviations.Both signals use the generator and its EMA copy, without an external feature encoder.Source: §4 · p. 6, §4.2 · p. 7

4.1 DMD Loss

• The internal reference and model velocity branches are separated by negating the condition vector, and their pointwise difference supplies the DMD correction.The interpretation assumes both branches approximate their respective marginal velocities.Source: §4.1 · p. 6
• The same DMD correction supervises velocity and clean-sample endpoint predictions using detached targets constructed from a re-noised clean-sample prediction.At t = 1, the endpoint prediction equals the one-step generator output; intermediate endpoint predictions need not follow its output law.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 Feature-Moment Loss

• The feature-moment loss compares pooled means and per-coordinate standard deviations in the generator EMA feature space, rather than matching the full feature distribution.The feature map mean-pools last-layer activations; the objective does not estimate the sample-space Gaussian component.Source: §4.2 · p. 7

4.3 Training Objective

• The training objective uses RCGM as Lbase and refreshes detached model/reference feature queues while updating the generator EMA copy.The queues approximate current population statistics from earlier model and EMA states.Source: §4.3 · p. 8, §4.2 · p. 8

5.1 Experimental Setup

• The setup evaluates full-parameter and rank-32/64 LoRA tuning on Z-Image-Turbo 6B and Qwen-Image 20B, using multiple prompt-following, quality, and world-knowledge benchmarks.LongText-CN and LongText-EN measure Chinese and English long-text rendering; checkpoints are evaluated at 1 and 2 NFE without changing training loss weights.Source: §5.1 · p. 8

5.2 Main Comparison

• At 2 NFE, IDM (Z-Image) scores 94.25% on LongText-CN and 90.20% on LongText-EN, the highest among compared methods at this budget.These exceed Z-Image-Turbo++ by 2.63 and 0.32 percentage points, respectively.Source: §5.2 · p. 9
• At 2 NFE, full-tuning IDM reaches 0.78 GenEval on Z-Image and 0.91 on Qwen-Image; rank-64 LoRA Qwen-Image reaches 0.90 while training 0.4B parameters.On Qwen-Image with full tuning, IDM and APEX have latencies of 0.44 and 0.45 seconds, respectively.Source: §5.2 · p. 9

5.3 Key Ablations

• Generator EMA features reach 77.81 GenEval versus 77.75 with DINOv2, and LongText-CN scores are 94.25 and 94.13, respectively.Both feature choices improve all five metrics over the internal DMD control; EMA features require no external feature encoder.Source: §5.3 · p. 11, Table 5 · p. 12
• Adding Lpop to internal DMD raises GenEval from 77.46 to 77.81, LongText-CN from 93.87 to 94.25, and LongText-EN from 89.79 to 90.20.Removing DMD decreases GenEval by 0.91 points, compared with 0.35 points when moment matching is removed.Source: §5.3 · p. 11, Table 5 · p. 12
• Equal-weight v+x regression reaches 77.81 GenEval, versus 76.64 for v-only and 77.26 for x-only; equal weights lead on the five aggregate metrics.The v-only and x-only controls retain Lpop, and the mixed losses reweight the same detached correction.Source: §5.3 · pp. 11–12, Table 5 · p. 12
• IDM scores 90.20 on LongText-EN versus APEX’s 88.76, while APEX leads on DPG-Bench, 88.64 versus 86.82.The published APEX reference uses a smaller training collection, so this comparison reflects complete training settings.Source: §5.3 · p. 12

5.4 Qualitative Results

• In Figure 5, full IDM more consistently preserves dense menu text, while the v+x control without Lpop produces more saturated colors in the sushi poster.The menu includes small Chinese and English names, numbers, and currency symbols in a compact layout.Source: §5.3 · p. 12, Figure 5 · p. 13

6 Conclusion

• 94.25% on LongText-CN and 90.20% on LongText-EN are IDM’s highest scores among compared two-NFE methods on Z-Image-Turbo 6B.The conclusion also reports experiments on backbones up to 20B parameters and controlled-ablation gains from EMA feature moments, with performance close to DINOv2.Source: §6 · p. 16

A Related Work

• IDM differs from trajectory-based and other distributional fast-generation methods by studying distributional supervision from the generator itself.The related-work overview also situates progressive distillation, consistency, shortcut, MeanFlow, adversarial, and score-based approaches.Source: A Related Work · p. 23
• TwinFlow and APEX estimate score differences with internally separated velocity branches; IDM adds a feature-moment loss to internal DMD.DMD uses a reference network and online fake-score model, while DMD2 adds adversarial supervision and stability techniques.Source: A Related Work · p. 23
• Unlike feature-space approaches including FD-Loss and Teacher-Feature Drifting, IDM matches feature means and per-coordinate standard deviations in the generator’s EMA feature space.The passage also discusses moment or kernel discrepancies, full-covariance Gaussian feature discrepancy, frozen features, and kernel or transport interactions.Source: A Related Work · p. 23
• IDM’s EMA representation reuses internal features that evolve with the generator, and its utility is evaluated against external, teacher, and live self-feature alternatives.The passage connects this motivation to Gaussian-plus-residual score structure and prior external representation-alignment approaches.Source: A Related Work · p. 23
• IDM focuses on LongText-Bench performance at 2 NFE, a setting used to assess extended Chinese and English text rendering.The related-work passage describes LongText-Bench as covering multiple scenarios and notes Z-Image-Turbo++’s use of LongText as a text-quality measure under limited sampling.Source: A Related Work · p. 23

B.1.1 First variation form

• Under the stated first-variation assumptions, the sample-space distribution gradient is the spatial gradient of the first-variation potential.Clean-distribution results assume smooth, strictly positive densities on a common domain; singular clean laws invalidate that interpretation, but do not by themselves invalidate noisy score fields or feature-moment objectives.Source: §B.1.1 · pp. 24–25

B.1.3 Important special cases

• For reverse KL, the ideal field is sθ − sr; practical DMD estimates its noisy-marginal counterpart, and also uses LPIPS regression on pre-generated teacher pairs.The finite-training online fake-score estimate can be lagged or biased.Source: §B.1.3 · p. 27
• The saturating GAN field agrees with the scaled-JSD distribution gradient, whereas the non-saturating GAN field is a detached optimal-discriminator update.The scaled-JSD field is weighted by the optimal real-probability discriminator.Source: §B.1.3 · pp. 27–28

B.2.1 Score Decomposition Identity

• The score of any smooth distribution decomposes into its moment-matched Gaussian score plus a non-Gaussian score residual.The Gaussian is defined by the distribution’s mean and covariance.Source: §B.2.1 · p. 28

B.2.2 Orthogonality of the Residual

• With the stated integrability and boundary conditions, the non-Gaussian score residual is orthogonal to constants and centered coordinates.Finite Fisher information together with a finite second moment suffices for the stated integrability condition.Source: §B.2.2 · p. 29

B.2.3 Reverse-KL Field Decomposition

• The reverse-KL field splits exactly into a Gaussian moment component and a non-Gaussian residual difference; this decomposition is unweighted, since general f-divergence weighting need not preserve residual orthogonality.The Gaussian component depends only on the first two moments, while the residual captures the remaining score structure.Source: §B.2.3 · p. 30

B.2.4 Diagonal Form and Connection to Estimators

• In a fixed coordinate space, a zero diagonal Gaussian moment-matching field requires matching the coordinate means and standard deviations; the feature-moment gradient instead defines a pooled feature-space regularizer.The feature-space regularizer is motivated by the Gaussian component of the reverse-KL field, not an estimate of that field.Source: §B.2.4 · pp. 30–31
• The DMD velocity difference estimates the full noisy reverse-KL field, while the feature-moment loss separately constrains pooled statistics; lower empirical gradient variance is not implied.The field estimate includes both moment and residual components.Source: §B.2.4 · p. 31

B.2.5 Variational characterization via L2 projection

• The moment-matched Gaussian score is the unique optimal affine approximation to the score in L2(p), and the residual is orthogonal to affine functions.The result assumes regularity conditions and finite Fisher information sufficient to make the optimization well-posed.Source: §B.2.5 · p. 31
• The score of the moment-matched Gaussian is the orthogonal projection of the distribution’s score, yielding a Pythagorean energy decomposition under that distribution.This projection property holds separately for each distribution, not automatically for their difference field.Source: §B.2.5 · pp. 31–32

B.3 Score–Velocity Identity

• For the linear Gaussian path, the marginal velocity field is the conditional expectation of the target velocity, and the section states an identity relating it to the marginal score.The proof proceeds through a Gaussian observation model and conditional expectations; at t = 1, the identity reduces to the standard normal score.Source: §B.3 · pp. 32–33
• The marginal velocity difference for two noisy distributions recovers their score difference up to the stated time-dependent proportionality factor.The result applies where the marginal fields and scores are well-defined at the same noisy input and time.Source: §B.3 · p. 33

B.4 DMD Loss: Score-Difference Identity and Regression

• The exact difference of marginal velocities is proportional to the noisy-marginal reverse-KL field at the same noisy input and time.The identity assumes both fields recover their respective exact marginal velocities.Source: §B.4 · p. 34
• In practice, the internally separated velocity-branch difference approximates the exact marginal-velocity correction rather than guaranteeing it.The passage specifies APEX-style condition separation and evaluates the correction at a stop-gradient prediction.Source: §B.4 · p. 34
• With detached targets, the endpoint-loss gradient is t^2 times the velocity-loss gradient, so combining the squared losses changes time weighting and update magnitude.The losses are collinear per-example under the stated calculation; at t = 1, the two losses coincide.Source: §B.4 · p. 34
• With approximate branches and correction processing, the equal-weight DMD regression gradient does not generally equal the unweighted time-averaged noisy reverse-KL gradient.An on-policy KL interpretation also requires the queried prediction law to agree with the law used to train the model-velocity branch.Source: §B.4 · p. 34

• The auxiliary regression uses the one-step output law pθ, with independent conditioning, noise, and time samples.Source: §B.5 · p. 35
• Gradients pass through the model-velocity branch, while generated samples and regression targets are detached.Source: §B.5 · p. 35
• Because the branches share θ, the auxiliary regression also updates shared parameters.Source: §B.5 · p. 35

B.6 Gradient of the Feature-Moment Loss

• With EMA feature-map parameters and queued features held fixed, derivatives flow through current samples, preventing direct within-update feature-map collapse but not slow degeneration.Source: §B.6 · p. 35
• Current model features retain gradients, while queued reference features are precomputed, detached, and used to define empirical feature moments.Source: §B.6 · p. 35
• The derived gradient is exact for the empirical objective at fixed queues, but is not generally an unbiased estimate of the population-loss gradient.Source: §B.6 · p. 35
• For a stabilized objective, the scalar stabilizer replaces each standard deviation in the gradient, including its denominator; the displayed gradient assumes positive variance and is unstabilized.Source: §B.6 · p. 36

B.7 Covariance Contraction Under Regression

• Tr Cov(m(Y)) < Tr Cov(X) whenever E[Tr Cov(X | Y)] > 0, for the optimal squared-error predictor m(Y) = E[X | Y].Source: §B.7 · p. 37
• The law of total covariance decomposes Cov(X) into E[Cov(X | Y)] plus Cov(m(Y)), yielding the covariance inequality.Source: §B.7 · p. 37
• The result applies only when the predictor realizes the conditional mean under the chosen coupling; it motivates possible variance suppression in ambiguous endpoint regression, not general underestimation by one-step generators.Source: §B.7 · p. 37
• Under an approximately independent, stationary-feature idealization with finite moments, mean-estimation variances scale as O(1/Nϕ) and O(1/K), but this does not establish lower empirical parameter-gradient variance, and queue staleness can introduce bias.Source: §B.7 · p. 37