PaperlayerView exact revision on arXiv

Source-linked AI summary

No Channels Left Behind: Understanding Diffusion Acceleration through Residual Gates

Junha Park, Jaemyung Yu, Jaehui Hwang, Jong-Seok Lee, Dongyoon Han

arXiv:2610.40914v1cs.LGcs.CV

TL;DR

The paper asks whether diffusion-training acceleration reflects only the representation structures targeted by auxiliary objectives or also changes to residual gating. It analyzes gate organization and introduces a teacher-free radial objective; Radial improves FID over Dispersive while increasing gate participation in the reported ImageNet setting.

  • Problem

    It is unclear whether diffusion-transformer acceleration comes from the representational structures explicitly imposed by auxiliary objectives or also from changes to the residual gating pathway.

    Source: §1 · p. 1

  • Method

    The paper studies class-conditioned AdaLN-Zero residual gates, measures their channel-wise organization with nPR, and proposes a teacher-free Radial objective that penalizes per-sample feature norms.

    Source: §1 · pp. 1–2, §3.2.2 · p. 5

  • Results

    Radial achieves FID 11.83 versus 13.19 for Dispersive on ImageNet 256×256 with SiT-B/2 at 400K steps, while mean final-block MLP gate participation rises from 0.434 in Vanilla to 0.642.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Gate participation tracks generation quality across REPA teacher and projector sweeps, and the results identify residual-gate organization as a measurable axis of diffusion acceleration.

    Source: §7 · p. 9

    Gate participation’s association with FID is not causal evidence: uniformizing interventions do not consistently improve quality, and Radial also affects other model parameters.

    Source: §5 · p. 9

Abstract

from arXiv · show

Training acceleration methods for diffusion transformers, such as REPA and dispersive regularization, are commonly explained by additional supervision, spatial-structure priors, or improved feature geometry. We revisit this view and uncover a striking internal change: the AdaLN-Zero residual gate $α$, which directly controls how conditioned updates enter the residual stream, is substantially reorganized by these methods. This observation motivates our hypothesis that acceleration arises, at least in part, from reshaping the residual gating pathway rather than solely from the feature structures explicitly imposed by auxiliary objectives. Supporting this view, accelerated models are markedly more sensitive to perturbations of $α$, and their gate geometry reflects semantic relationships among classes. We further introduce the normalized participation ratio (nPR) to characterize the channel-wise organization of the gates and find that, across REPA teachers, projector architectures, and radial-objective targets, broader gate participation accompanies better generation quality. Guided by this insight, we propose a simple teacher-free radial objective that promotes broader gate participation through feature-norm regularization, without pairwise feature comparisons. On ImageNet $256\times256$ with SiT-B/2, our method improves FID over dispersive loss while increasing mean gate participation over timesteps and classes, further supporting the link between gate organization and training acceleration. Our results identify residual-gate reorganization as a measurable axis of diffusion training acceleration and suggest a simple direction for further acceleration.

1 INTRODUCTION

The introduction asks whether auxiliary objectives accelerate diffusion transformers by changing explicitly supervised representations or by reorganizing the residual gating pathway. It connects gate organization to generation quality and proposes a teacher-free radial objective to promote broader participation.

1 INTRODUCTION

  • The authors ask whether faster training comes from the representational structures imposed by auxiliary objectives or from changes to a more fundamental transformer mechanism.They frame this question against teacher-guided alignment, spatially structured alignment, and feature-geometry regularization.Source: §1 · p. 1
  • Because AdaLN-Zero’s residual gate α scales each branch’s channel outputs before residual addition, the authors hypothesize that acceleration reorganizes how features enter the residual stream.The gate is shared across token positions.Source: §1 · p. 1
  • Gate distances increase with semantic distance between classes, while broader participation tracks better generation quality across REPA teachers and projector designs.Accelerated models are more vulnerable than the unaccelerated baseline to gate disruption; participation is measured across timesteps and classes.Source: §1 · p. 2
  • On ImageNet 256×256 with SiT-B/2 trained for 400K steps, Radial achieves FID 11.83 versus 13.19 for Dispersive and raises mean final-block MLP gate participation from 0.434 in Vanilla to 0.642.Radial also improves over Vanilla’s FID of 16.24.Source: §1 · p. 2
  • The proposed Radial objective penalizes deviations of each sample’s flattened feature norm from a target, without a pretrained teacher or pairwise feature comparisons.It increases gate participation and improves generation quality, supporting gate participation as a candidate mechanism underlying acceleration.Source: §1 · p. 2
  • The authors identify residual-gate organization as an overlooked axis of diffusion acceleration and a way to promote it without representation teachers or pairwise comparisons.Their conclusion is that auxiliary objectives also reorganize the residual gating pathway, whose channel participation tracks generation quality in controlled sweeps.Source: §1 · p. 2

2 BACKGROUND

This section situates AdaLN-Zero gating in diffusion transformers and distinguishes the residual gate from modulation of the branch input. It also summarizes teacher-guided and teacher-free approaches to accelerating training.

2 BACKGROUND

  • Diffusion transformer blocks add self-attention and MLP updates to the residual stream, and this section focuses on their AdaLN-Zero conditioning.SiT adopts the DiT transformer backbone for interpolant-based generative modeling.Source: §2 · p. 2
  • The conditioning pathway produces scale, shift, and gate vectors; α directly scales each branch output before residual addition, unlike β and γ, which modulate its input.These channel-wise modulation vectors are broadcast across tokens, and the attention and MLP branches have separate parameters in each block.Source: §2 · pp. 2–3
  • The paper focuses on how learned branch-specific residual gates allocate contributions across channels and organize across training objectives.The authors describe this organization under accelerated training as underexplored relative to prior analyses of AdaLN initialization and conditional embeddings.Source: §2 · p. 3
  • Prior acceleration losses include teacher-guided REPA and iREPA, which align diffusion features to frozen vision-model representations, and teacher-free dispersive loss, which enlarges within-batch feature distances.REPA and iREPA use a learnable projector; Vanilla refers to the baseline SiT model.Source: §2 · pp. 2–3

3 UNDERSTANDING ACCELERATION IN DIFFUSION TRANSFORMERS

The section tests whether auxiliary objectives reshape AdaLN-Zero gating, then examines semantic structure and whether channel-wise gate participation tracks generation quality.

3.1 MOTIVATION

  • Removing AdaLN-Zero modulation weakened both acceleration gains: REPA’s FID change shifted from −18.24 to −13.48, while Dispersive’s shifted from −4.98 to +1.15.The comparison replaced AdaLN-Zero with in-context conditioning, removing the modulation pathway including residual gates.Source: §3.1 · p. 3, Table 1 · p. 3
  • At matched δ = 0.30, REPA gate perturbations changed FID by up to +5.356, while removing AdaLN-Zero substantially weakened auxiliary-objective gains.The test-time intervention perturbed frozen models; it measures sensitivity to departures from learned gates, not preference for uniform gates.Source: §3.1 · pp. 3–4, Table 1 · p. 3, Figure 2 · p. 4

3.2.1 HOW DO RESIDUAL GATES BEHAVE IN TRAINING ACCELERATION METHODS?

  • At WordNet distance 22, Euclidean gate distances reached 1.678×, 1.217×, and 1.428× starting values for REPA, Vanilla, and Dispersive, respectively.Centered cosine distances also rose by 0.574, 0.225, and 0.441, respectively; gate distances generally increased with semantic distance in all three models.Source: Figure 4 · p. 4, §3.2.1 · p. 5

3.2.2 DEFINING GATE PARTICIPATION AND ITS IMPACT

  • nPR(α) measures how evenly squared gate magnitude is distributed across channels, ranging from 1/D to 1 and invariant to uniform nonzero rescaling.Equal nonzero magnitudes on m of D channels give nPR(α) = m/D; the measure reflects allocation rather than overall gate scale.Source: §3.2.2 · p. 5
  • Across eight REPA teachers, mean gate participation correlated with FID at r = −0.962; across four projector variants, the correlation was r = −0.991.The projector sweep used a fixed teacher, target preprocessing, alignment weight, and training budget.Source: Figure 6 · p. 6, §3.2.2 · p. 6, Table 2 · p. 6

4 INCREASING GATE PARTICIPATION FOR BETTER GENERATION QUALITY

The section introduces a teacher-free feature-norm objective intended to broaden residual-gate participation, then tests its relationship to generation quality. In the reported SiT-B/2 comparison, Radial improves FID over Vanilla and Dispersive while attaining the highest mean gate participation.

The radial objective

  • The Radial objective adds a per-sample penalty on the flattened final-block feature norm, using target c and weight λ, without a teacher or pairwise sample distances.The target sweep selects the value with the lowest observed FID for the performance comparison.Source: §4 · p. 7

Promoting gate participation with our radial objective

  • Under channel symmetry and regularity conditions, a sufficiently small conditional gate update can reduce gate concentration through input-dependent inverse-norm weighting.The analysis excludes the skip–residual cross term and does not guarantee participation growth under the full optimizer.Source: §4 · p. 7

Higher participation and lower FID under radial training

  • Across five target settings, mean L12 MLP gate participation and FID correlate at r = −0.916; FID falls from 45.72 to 39.43 and participation rises from 0.381 to 0.577 at the selected target k = 10.The target was selected by the lowest observed FID.Source: §4 · pp. 7–8

Generation quality with and without a teacher

  • At 400K steps, Radial reaches FID 11.83 versus 16.24 for Vanilla and 13.19 for Dispersive, with mean L12 MLP gate participation of 0.642 versus 0.434 and 0.525, respectively.The table compares SiT-B/2 at 256 × 256; Radial is teacher-free.Source: §4 · p. 8, Table 3 · p. 8

Gate participation during training

  • Radial maintains higher gate participation than Vanilla at 50K, 100K, 200K, and 400K steps; Dispersive also exceeds Vanilla, peaking at 200K.Measurements use EMA weights and average over timesteps and classes.Source: §4 · p. 8

5 DISCUSSION AND LIMITATIONS

The discussion interprets projector comparisons narrowly and distinguishes observed associations from established causal mechanisms. The tested results do not show that spatial mixing is required, while the proposed participation mechanism has explicit limits.

5 DISCUSSION AND LIMITATIONS

  • In the tested configuration, neighboring-token mixing inside the projector is not required to attain comparable performance.The 1×1 projector reaches FID 51.33 versus 51.68 for an additional 3 × 3 convolutional projector under the same teacher normalization and alignment settings.Source: §5 · pp. 8–9
  • The lower FID of the deep linear head than the nonlinear MLP motivates testing projector nonlinearities, without establishing a general advantage for linear heads.The discussion frames this as motivation for further examination.Source: §5 · p. 9
  • Participation–FID associations do not establish causation; Radial affects gates indirectly, and its local-gradient result does not guarantee participation growth under the full optimizer.The proposition also does not establish that participation growth causes the FID improvement.Source: §5 · p. 9

6 RELATED WORK

Prior work accelerates diffusion transformers through teacher-guided alignment, spatial structure, or feature-geometry objectives. This paper situates its analysis around the learned residual gate and its channel allocation rather than claiming that spatial information is necessary.

6 RELATED WORK

  • REPA aligns intermediate diffusion features with pretrained visual representations, while iREPA and HASTE examine alignment targets, projectors, and timing or scope of supervision (Yu et al., 2025; Singh et al., 2026; Wang et al., 2025).The paper analyzes internal modulation induced by these training choices, focusing on residual-gate participation and geometry.Source: §6 · p. 9
  • iREPA and HASTE use spatial structure in different ways, while the paper's projector experiments do not claim spatial information is necessary in diffusion models (Singh et al., 2026; Wang et al., 2025).The experiments examine projection architecture within tested alignment configurations.Source: §6 · p. 9
  • Related modulation analyses study initialization and conditional embeddings, whereas this work focuses on learned branch-specific residual gate α and its channel allocation and geometry (Zhu et al., 2026; Pham et al., 2026).DiT uses AdaLN-Zero and SiT adopts the DiT backbone for interpolant-based modeling (Peebles & Xie, 2023; Ma et al., 2024).Source: §6 · p. 9
  • The paper applies participation ratio to squared gate coordinates to measure magnitude allocation across channels, not hidden-representation spectral rank or intrinsic dimension.It is used as a diagnostic associated with generation quality.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents residual-gate participation as a lightweight quantity associated with generation quality and reports improved FID for teacher-free Radial over Dispersive. The accompanying statements describe AI assistance and point to appendices for implementation details.

7 CONCLUSION

  • Accelerated models depend more strongly on residual gates, whose participation tracks FID across teacher and projector sweeps; teacher-free Radial raises participation and improves FID over Dispersive.Gate participation can be computed from the conditioning pathway without an image forward pass.Source: §7 · p. 9

AI USE STATEMENT

  • The authors report using generative AI for small coding tasks, proof assistance, mathematical-claim formulation, related-work search, readability editing, and formatting checks, while stating it did not originate the core methodology.They report reviewing the AI-assisted work and taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

A.1 MODELS AND TRAINING

The appendix specifies the SiT architectures and training setup, then defines how gate participation is aggregated across timesteps and classes.

A.1 MODELS AND TRAINING

  • SiT-XL/2 analyses use 28 blocks, width 1,152, and 16 heads, while SiT-B/2 projector and Radial experiments use 12 blocks, width 768, and 12 heads.Both use 2 × 2 latent patches and an MLP expansion ratio of four; layer numbering is one-based.Source: §A.1 · p. 12
  • SiT-B/2 runs use seed 42, batch size 256, bfloat16, and AdamW at 10−4; Dispersive uses final-block InfoNCE-L2 with coefficient 0.25 and temperature 1.The runs use a linear interpolation velocity path and cached ImageNet latents from the ft-MSE VAE.Source: §A.1 · p. 12

A.2 GATE PARTICIPATION MEASUREMENT

  • Gate participation is evaluated over 101 evenly spaced timesteps and all 1,000 ImageNet classes, averaging participation values across 101,000 conditions.Participation values—not gate vectors—are averaged, and MHSA and MLP gates are not concatenated.Source: §A.2 · p. 12

A.3 FID EVALUATION

The local FID protocol evaluates 50,000 generated ImageNet images with a specified evaluator, sampler, and decoding setup.

A.3 FID EVALUATION

  • Local FID uses the ADM evaluator and ImageNet-256 reference statistics on 50,000 generated images, with EMA weights and the sd-vae-ft-ema decoder.Sampling uses CFG 1.5 and seed 42.Source: §A.3 · p. 12
  • The sampler is adaptive Dormand–Prince (dopri5), with absolute tolerance 10−6 and relative tolerance 10−3.Source: §A.3 · p. 12
  • The evaluator reports FID, sFID, inception score, precision, and recall; gate-intervention and teacher-sweep pilots use separate protocols.Source: §A.3 · p. 12

B DETAILS OF THE GATE ANALYSIS

These appendices detail the conditioning and gate-intervention pilots, gate-structure analyses, and projector sweep, including their evaluation settings.

B.1 IN-CONTEXT CONDITIONING PILOT

  • The in-context pilot compares six SiT-B/2 runs trained for 100K steps, varying only the auxiliary objective within each conditioning architecture.REPA aligns L4 with DINOv2-B through a three-layer MLP projector; Dispersive is applied at L12.Source: §B.1 · p. 12

B.2 GATE INTERVENTION PILOT

  • The intervention pilot modifies the realized L8 MLP gate during sampling, redistributing squared magnitudes toward uniform or concentrated profiles while matching relative perturbation size.The changes preserve gate signs, norm, and channel ordering; evaluations use 50,000 images, CFG 1.0, and 50-step Euler sampling.Source: §B.2 · p. 13

B.3 GATE HISTOGRAMS AND WORDNET DISTANCES

  • Gate histograms pool signed L8 MLP entries across 1,000 classes, while WordNet analysis compares class-conditioned gate magnitudes across semantic-distance bins.The analyses use SiT-XL/2 EMA checkpoints at 400K steps and t = 0.5; Euclidean and centered-cosine distances are normalized relative to the WordNet-distance-2 bin.Source: §B.3 · p. 13

B.4 GATE STRUCTURE AT ADDITIONAL TIMESTEPS

  • At t = 0.25 and 0.75, gate distances still increase with WordNet distance, with REPA showing the largest separation across semantic distances.Differences in gate scale among Vanilla, Dispersive, and REPA also persist at both timesteps.Source: §B.4 · p. 13

B.5 TEACHER SWEEP

  • The teacher sweep measures L8 MLP gate nPR for released REPA checkpoints and uses their released FIDs, evaluated with ADM, CFG 1.0, and 250-step SDE sampling.The additional-timestep figures show WordNet-distance curves and signed gate histograms for Vanilla, Dispersive, and REPA.Source: §B.5 · pp. 14–15

B.6 PROJECTOR SWEEP

  • The projector sweep trains SiT-B/2 with DINOv2-B alignment at L4 for 50K steps, comparing four heads from single linear to three-layer MLP.The heads vary in depth and nonlinearity, with parameter counts from 590,592 to 2,624,256; teacher targets are normalized within each image and channel.Source: §B.6 · p. 15

C.1 TARGET SWEEP AND TARGET CHOICE

The appendix selects a feature-norm target using FID and gate participation, then gives performance and training-dynamics protocols and an analytical condition on gate updates.

C.1 TARGET SWEEP AND TARGET CHOICE

  • FID is 39.43 at the selected target c = 4500, versus 45.41 for Vanilla, while L12 MLP gate participation reaches 0.577.The target c = 3503.54 yields FID 40.02; the largest tested target, 6000, yields 39.44 and participation 0.564.Source: §C.1 · p. 15, Figure C · p. 15

C.2 PERFORMANCE COMPARISON

  • The C.2 comparison uses 400K-step SiT-B/2 checkpoints at 256×256, with Radial and Dispersive teacher-free runs also shown in Fig. 7.Radial uses target c = 10ND ≈ 4434 and coefficient 5 × 10−8; Dispersive uses coefficient 0.25 at the final block, and REPA aligns L4 with DINOv2-B.Source: §C.2 · pp. 15–16

C.3 TRAINING DYNAMICS

  • Fig. 7 tracks normalized auxiliary losses and L12 MLP gate participation at 50K, 100K, 200K, and 400K training steps.The three trajectories are from the 400K SiT-B/2 runs; the auxiliary losses are normalized from their initial to minimum values.Source: §C.3 · p. 16

D CONDITIONS AND PROOF OF PROPOSITION 1

  • Under the stated analytical setting, the gate update does not decrease nPR for sufficiently small steps, and a positive directional derivative implies a strict increase.When the derivative is zero, a common rescaling preserves nPR; the argument concerns sufficiently small steps, not arbitrary learning rates.Source: D CONDITIONS AND PROOF OF PROPOSITION 1 · p. 17

D.1 SCOPE OF THE ANALYSIS

The analysis narrows its claims by omitting the skip–residual cross-term and considering a local Euclidean gate update. It does not establish corresponding claims for the full residual norm or full training dynamics.

D.1 SCOPE OF THE ANALYSIS

  • The analysis omits the gate-weighted cross-term; orthogonality or a small cross-term value at one gate does not show it vanishes or has a small gate derivative.The proposition therefore concerns the cross-term-excluded norm rather than the full residual norm without additional assumptions.Source: §D.1 · p. 18
  • The proposition applies to the cross-term-excluded norm, not the full residual norm without additional assumptions.Source: §D.1 · p. 18
  • The local Euclidean gate update does not imply monotonic nPR growth or a causal relationship between nPR and FID under full training dynamics.Gate parameterization, other features, task gradients, and optimizer state can also change during training.Source: §D.1 · p. 18
Loading 2610.40914v1…