No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves

1 Introduction

• Prior methods use an external representation task or pretrained encoder for guidance, motivating the question of whether diffusion transformers can provide guidance without either component.The external-encoder route is less usable when a suitable encoder is unavailable.Source: §1 · p. 2
• The authors observe that diffusion-transformer features become more refined as layers increase and noise decreases, while the model learns discriminative representations during generative training.Representation accuracy peaks around layer 20 and then drops as the model shifts toward generating high-frequency image details.Source: §1 · p. 2
• SRA aligns an earlier-layer representation at higher noise with a later-layer representation at lower noise, using an EMA teacher and a student projection head.The authors describe the method as requiring no external representation component or architectural modification.Source: §1 · pp. 2–3
• SRA brings performance improvements to both DiTs and SiTs, with sustained training-speed acceleration and nontrivial generation improvement.The authors also report that ablations highlight the role of internal representations in SRA’s success.Source: §1 · p. 3

2.1 Preliminary: Training Object of DiT and SiT

• Denoising models learn to reverse a process that adds noise to data, predicting the added noise with a mean-squared-error objective.The paper uses the generative loss label to represent the denoising and flow-based training objectives uniformly.Source: §2.1 · p. 3
• Flow-based models learn a velocity field for a probability-flow ODE, with noisy data formed by mixing data and Gaussian noise using time-dependent coefficients.The model minimizes a loss to learn the velocity field.Source: §2.1 · p. 3

2.2 Self-Representation Alignment

• SRA aligns a student’s earlier-layer output at time t with a teacher’s later-layer output at t−k, where the teacher input has lower noise.A lightweight trainable MLP projection transforms the student output; its head can be discarded after training, so SRA need not alter the diffusion-transformer architecture.Source: §2.2 · p. 4
• SRA minimizes patch-wise distance between the teacher output and projected student output, then combines this alignment objective with the generative objective.The hyperparameter λ controls the trade-off between the two objectives.Source: §2.2 · p. 4

2.3 Teacher Network

• The teacher is formed from past student iterations using exponential moving average updates, with α = 0.9999 unchanged reported to work well.The authors report that training was stable without clustering constraints, batch normalization, or centering.Source: §2.3 · pp. 4–5

3.1 Experimental Setup

• Experiments follow DiT and SiT training setups, report FID, sFID, IS, precision, and recall, and use the ADMs evaluation suite with 50K samples.The comparison covers pixel diffusion, latent diffusion with U-Nets, and latent diffusion with transformers.Source: §3.1 · p. 5, Methods for Comparison · p. 5

3.2 Component-Wise Analysis

• Component-wise tests favor late teacher layers guiding early student layers, lower-noise teacher features with an interval of 0.1 or mean 0.1, and dynamic intervals of 0–0.2.The authors also choose λ = 0.2 and retain a projection head, which performs better than direct alignment.Source: §3.2 · pp. 5–6, Table 1 · p. 6

3.3 System-Level Comparison

• At 800 epochs, SiT-XL + SRA reaches FID 1.58 and IS 311.4 on ImageNet 256×256, outperforming methods such as MaskDiT and matching methods such as REPA.The authors also report that it outperforms vanilla SiT-XL with 1,000 fewer epochs and improves further with longer training.Source: §3.3 · p. 8, Table 3 · p. 8
• At higher resolution, SRA surpasses its baseline with 3× fewer training iterations; on COCO2014, MMDiT + SRA improves FID from 5.66 to 4.75 and PickScore from 20.65 to 21.14.With the same iterations at higher resolution, SRA outperforms REPA on sFID, IS, and Rec, and is on par on other metrics.Source: §3.3 · pp. 7–8, Table 2 · p. 7, Table 4 · p. 8
• SRA’s improvement over REPA is more sustained: REPA’s performance saturates after about 200 epochs, while SRA’s boost continues without external models.The authors attribute SRA’s sustained boost to the teacher’s progressively improving representation guidance.Source: §3.3 · pp. 6–9, Figure 6 · p. 8

3.4 Ablation Study

• SRA improves SiT’s linear-probing results across blocks and timesteps, and linear-probing accuracy strongly correlates with FID as the teacher alignment layer varies.The latter comparison varies teacher layers used for alignment.Source: §3.4 · p. 9
• The teacher’s representation quality rises from 38.1 at 200K iterations to 54.2 at 800K and remains above the student’s at each stage.The authors present this as evidence that the teacher’s improving capacity provides better guidance as training proceeds.Source: §3.4 · p. 9

4 Related Work

• MaskDiT and SD-DiT add MAE or IBOT tasks, while TREAD uses token routing with an MAE loss.These approaches introduce external representation-learning tasks into diffusion training.Source: §4 · p. 9
• REPA uses a large-scale pretrained representation model, while VA-VAE and MAETok align tokenizer latents with external foundation-model representations.Source: §4 · p. 9
• Unlike these approaches, this work seeks representation guidance from the diffusion model itself and its training paradigm.Source: §4 · p. 9

5 Limitation and Future Work

• Limited computational resources prevented large-scale text-to-video pretraining with SRA.Source: §5 · p. 10
• The authors argue video use is conceptually sound, noting image-domain performance comparable to REPA and the absence of a comparably strong open-domain video encoder.Source: §5 · p. 10
• They point to transferable representations in pretrained text-to-video models as support for potential video applications and identify theoretical analysis as future work.Source: §5 · p. 10

6 Conclusion

• SRA aligns earlier-layer, higher-noise latents with later-layer, lower-noise latents to improve representation learning and generation without external components.Source: §6 · p. 10

A Investigation of Representations in DiT

• DiT representations follow a coarse-to-fine trend as block depth increases and noise decreases, similarly to SiT.Source: A Investigation of Representations in… · p. 16, Figure 8 · p. 16

B Descriptions for Two Types of Baseline Models

• The paper uses DiT and SiT as baselines, describing them as denoising-autoencoder and stochastic-interpolant model variants, respectively.Source: B Descriptions for Two Types of… · p. 16

B.1 Denoising Diffusion Probabilistic Models (DiT)

• DDPMs define a forward process that adds Gaussian noise over timesteps and learn its reverse; sampling starts from Gaussian noise and iteratively recovers a data sample.Source: §B.1 · pp. 16–17
• The DDPM model uses a denoising autoencoder objective, and Improved-DDPM jointly learns the covariance with the noise-prediction output.Source: §B.1 · pp. 16–17

B.2 Stochastic Interpolants Models (SiT)

• SiT uses a continuous interpolation between data and Gaussian noise, modeled by a probability-flow ODE; its diffusion coefficient can be selected independently during sampling.Source: §B.2 · p. 17

C Hyperparameter and More Implementation Details

• Training uses mixed precision, gradient clipping, FusedAttention, precomputed stable-diffusion-VAE latents, a two-layer SiLU projection head, and smooth-L1 alignment.The authors report smooth-L1 performed slightly better than L2 and had smoother gradient transitions than L1.Source: C Hyperparameter and More… · p. 17
• Experiments use ImageNet at 256×256 or 512×512; DiT and SiT sampling each use 250 function evaluations with their specified samplers.Source: C Hyperparameter and More… · p. 18
• Linear probing follows REPA and I-DAE, training a linear layer for 80 epochs after pooling and batch normalization of model latents.Source: C Hyperparameter and More… · p. 18

D Evaluation Metric

• FID measures similarity between real and generated feature distributions, while sFID adds spatial information to assess structural fidelity.Source: D Evaluation Metric · pp. 18–19
• Inception Score evaluates generated-image quality and diversity using class probabilities; higher values indicate both.Source: D Evaluation Metric · p. 19
• Precision and recall measure the trade-off between generated-sample realism and coverage of the real-image manifold.Source: D Evaluation Metric · p. 19

E Methods for Comparison

• The comparison set also includes classifier-guided ADM, adaptive-noise VDM++ [41], high-resolution Simple Diffusion, cascaded CDM, and latent-space LDM.Source: E Methods for Comparison · p. 19
• DiT uses a pure transformer backbone with AdaIN-zero modules, while SiT studies continuous flow-based modeling as a training-efficiency alternative.Source: E Methods for Comparison · p. 19
• MaskDiT [91], TREAD [44], MAETok, and SD-DiT use auxiliary reconstruction or representation-learning objectives to guide diffusion-transformer training.MAETok aligns targets from HOG, DINOv2 [64], and CLIP; SD-DiT combines DINO and BEIT losses.Source: E Methods for Comparison · p. 19
• REPA aligns diffusion-model features with a large-scale pretrained representation model such as DINOv2 [64].Source: E Methods for Comparison · p. 19

F Teacher Network Updating Role

• Among teacher updates, student-copy initialization performs worst, a ramped momentum coefficient from 0.996 to 1 works poorly, and fixed α=0.9999 is suitable for SRA.The authors attribute the fixed high momentum’s suitability to a stable alignment target that avoids destabilizing the primary diffusion loss.Source: F Teacher Network Updating Role · p. 20

G Principle of our hyperparameter

• A lightweight projection head is intended to make alignment softer and reduce disruption to the model’s original generative behavior.Source: G Principle of our hyperparameter · p. 20
• The teacher block should provide strong semantics, while the student uses early layers; the teacher timestep should balance semantic quality against distance from the student timestep.The authors say these selection principles also apply to other diffusion transformers, including MMDiT.Source: G Principle of our hyperparameter · pp. 20–21
• λ is chosen to keep diffusion and alignment losses on comparable scales; too small weakens regularization, while too large lets alignment dominate and can harm sample quality.The stated example scales are approximately 0.7 for diffusion loss and 1.6 for alignment loss; λ=0.02 and λ=2 illustrate the two extremes.Source: G Principle of our hyperparameter · p. 21

• SRA improves model performance within the same training time despite an extra forward-only pass, using only 8/12 SiT-B blocks or 20/28 SiT-XL blocks for the target.The cited memory-use and per-epoch-time comparison uses total batch size 256 on one machine with 8 A100 GPUs.Source: H Training Speed and GPU Memory Usage… · p. 21, Table 7 · p. 21

• Figure 7c’s horizontal axis is linear-probe accuracy from an original SiT-XL/2 checkpoint trained for 7M iterations.Source: I Detailed Setup of Ablation Study · p. 21
• Figure 7c’s vertical axis is FID after 400K iterations of SRA training without classifier-free guidance.Source: I Detailed Setup of Ablation Study · p. 21
• The authors treat the experiment as a test of SRA because it uses no representation component and applies representation supervision only during generative training.Source: I Detailed Setup of Ablation Study · p. 21

J Ablation Results of DiT

• DiT with SRA shows a strong correlation between generative capability and representation guidance, matching the reported SiT pattern.Figure 9 presents the DiT analysis and is described as showing a similar tight coupling.Source: J Ablation Results of DiT · p. 22, Figure 9 · p. 22

K More Discussion on Related Work

• Unlike traditional distillation from a pretrained teacher, SRA uses an EMA teacher and an internal feature-alignment pretext task without requiring a pretrained model.The related-work discussion contrasts this with pretrained-model guidance and sampling-distillation methods such as CM and LCM [58].Source: K More Discussion on Related Work · pp. 22–23
• The discussion traces diffusion-transformer development from U-ViT and DiT to SiT, while positioning SRA as a training improvement for DiT and SiT.Source: K More Discussion on Related Work · p. 22
• Prior studies use diffusion-model representations for downstream tasks such as segmentation and detection, whereas this work uses them to enhance generation.The cited examples include GD [61], DDAE, I-DAE [14], Repfusion, DiffusionDet, and DreamTeacher.Source: K More Discussion on Related Work · p. 22
• This study focuses on class-conditional diffusion-transformer pretraining, unlike guidance methods for low-bias synthesis, unconditional generation, or attribute manipulation and interpolation.The cited examples are lbGen [38], RCG, and Diff-AE.Source: K More Discussion on Related Work · p. 23

L More Qualitative Results (CFG 4.0)

• Samples cover loggerhead turtle, sulphur-crested cockatoo, golden retriever, white fox, tiger, and red panda.Source: Figure 10 · p. 24, Figure 11 · p. 24, Figure 12 · p. 25
• Additional samples depict acoustic guitar, balloon, baseball, fire truck, laptop, and ice cream.Source: Figure 16 · p. 27, Figure 17 · p. 27, Figure 18 · p. 28
• The remaining sample classes are cheeseburger, cliff drop-off, coral reef, and lakeside.Source: Figure 22 · p. 30, Figure 23 · p. 30, Figure 24 · p. 31