PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Residual Connectivity Matters for Diffusion Transformers

Yuhe Liu, Xinyin Ma, Gongfan Fang, Songhua Liu, Xinchao Wang

arXiv:2609.33203v1cs.CV

TL;DR

DiTs use a uniform residual stream that conflates layer contributions, prompting the question of how to combine transformer scalability with structured connectivity. The paper introduces differentiable, stage-matched residual routing and reports improved FID across DiT scales, with up to 1.73× fewer matched-FID training steps.

  • Problem

    DiTs’ uniform additive residual stream conflates preceding layers, unlike U-Nets’ structured pathways, motivating the question of how to combine transformer scalability with structured connectivity.

    Source: §1 · pp. 1–2

  • Method

    The method collects differentiable encoder representations and routes them to stage-matched decoder layers for residual reconstruction.

    Source: §3.2 · p. 4, §5 · p. 9

  • Results

    Up to 1.73× fewer training steps are needed at matched FID, and FID improves across DiT-S/2, DiT-B/2, and DiT-XL/2 at 400K iterations.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The results identify adaptive cross-layer connectivity as an underexplored factor in DiTs and structured pathways as a direction for improving scalable generative models.

    Source: §1 · p. 2

    Validation on larger text-to-image and text-to-video models remains future work, and mirror routing may not be optimal with stronger multimodal conditioning or temporal dependencies.

    Source: E LIMITATIONS · p. 13

Abstract

from arXiv · show

Diffusion Transformers (DiTs) have established themselves as a scalable backbone for high-fidelity image synthesis. However, unlike U-Net based diffusion models that rely on rigid, hand-crafted skip connections, DiTs predominantly use a uniform residual stream that integrates all preceding layers as a monolithic state. In this work, we rethink residual connections in diffusion transformers and propose to transform them from passive summation into an active retrieval mechanism optimized for image denoising. First, we conduct a systematic analysis of DiT's internal representation, revealing a latent preference for early-layer feature reuse and symmetric layer guidance. Motivated by this, we introduce a structured connectivity design that explicitly integrates local residual connections with long-range pathways. Instead of static skip connections or dense all-layer routing, our method enables each transformer block to selectively ``attend'' to critical earlier representations, dynamically retrieving spatial and semantic cues through direct, differentiable cross-depth paths. Experiments show that our adaptive connectivity leads to faster convergence, with up to $1.73\times$ fewer training iterations, and significant gains in FID and visual quality with less than $0.1\%$ additional parameters, further improving a strong REPA-XL/2 model from $5.9$ to $4.34$ FID without guidance and reaching $1.39$ FID with classifier-free guidance. Our findings suggest that adaptive cross-layer connectivity is a critical yet underexplored factor in diffusion transformers, and that incorporating structured information pathways provides a simple and effective direction for improving scalable generative models.

1 INTRODUCTION

DiTs’ uniform residual stream contrasts with structured U-Net pathways, raising the question of how to combine transformer scalability with structured connectivity. The paper proposes adaptive cross-layer routing and reports faster convergence and improved FID.

1 INTRODUCTION

  • DiTs’ local additive updates conflate preceding layers into one hidden state, diluting individual layer contributions and imposing a rigid backward-pass topology.The authors contrast this with U-Nets’ structured pathways, which preserve multi-scale features.Source: §1 · p. 1
  • With 0.28M additional steps, REPA-XL/2 improves from 5.9 to 4.94 FID, versus 5.87 for continued fine-tuning; at 0.35M steps, it reaches 4.34 FID.The 0.28M-step comparison is shown in Figure 1; the 0.35M-step result is reported in the introduction.Source: Figure 1 · p. 2, §1 · p. 2
  • The paper asks how DiTs can combine transformer scalability with structured connectivity patterns that support generative modeling.Prior approaches use static projections for long skips or dense routing over all preceding layers.Source: §1 · p. 2
  • The proposed structured connectivity turns residual summation into active retrieval along cross-layer paths, allowing gradients to flow directly to stage-matched encoder layers.The method uses less than 0.1% additional parameters and reaches baseline FID with up to 1.73× fewer iterations.Source: §1 · p. 2
  • The authors identify adaptive cross-layer connectivity as an underexplored factor and structured information pathways as a direction for improving scalable generative models.The introduction also reports 1.39 FID with classifier-free guidance.Source: §1 · p. 2

2 RELATED WORK

Related work spans diffusion-model improvements and architectural approaches to information flow. The paper situates its routing design among structured and adaptive residual pathways.

2 RELATED WORK

  • Diffusion-model work has pursued likelihood and sampling improvements, non-Markovian processes, representation alignment, and architectural design.The cited overview also notes that guidance and architecture can affect generation quality.Source: §2 · p. 2
  • Figure 2 presents all, first, last, and mirror source-routing strategies; the final design uses sparse, stage-matched mirror paths for decoder-side reconstruction.Encoder-side representations are collected as differentiable sources and routed to later layers.Source: Figure 2 · p. 3
  • Prior architectural approaches include UNet skip connections, static long skips in U-ViT and U-DiTs, and DDT’s decoder conditioning on the final encoder output.The paper’s method instead lets decoder layers draw dynamically on stage-matched encoder representations.Source: §2 · p. 3

3 METHOD

The method replaces uniform residual aggregation with differentiable routing over structured cross-layer paths. Its design follows observed preferences for early-layer representations and mirrored layer pairs, then uses stage-matched mirror routing to reuse features.

3.1 PRELIMINARY: RESIDUALS AS DEPTH-WISE ROUTING

  • Standard residual updates implicitly accumulate prior layer outputs with fixed unit coefficients, leaving no mechanism to selectively emphasize useful intermediate representations.The method treats residual connections as information routing, not only as optimization shortcuts.Source: §3.1 · p. 3

3.2 OVERVIEW: DIFFERENTIABLE RESIDUAL COLLECTION AND ROUTING

  • Intermediate encoder and decoder representations, including the patch-embedding output, remain differentiable sources so decoder routing gradients can flow back to encoder features.The patch-level source provides access to representations before any transformer block.Source: §3.2 · p. 4

3.3 RESIDUAL ROUTING OPERATOR

  • The routing operator reconstructs each sublayer input from the current representation and selected sources before applying the sublayer transformation.It is inserted before both self-attention and MLP, allowing selected features to augment the identity stream.Source: §3.3 · pp. 4–5, Figure 3 · p. 4

3.4 ROUTING ON ALL LAYERS

  • DiTs assign high importance to early-layer representations and show a preference for mirrored or symmetric layer pairs when routing across layers.These observations motivate encoder-side source reuse and symmetric pathways in the proposed design.Source: §3.4 · p. 5, Figure 4 · p. 5

3.5 SOURCE ROUTING STRATEGIES

  • Mirror routing achieves the best FID among the tested source strategies by giving each decoder layer a sparse, stage-matched encoder source.Unlike dense all routing, it supplies the correspondence directly, reducing routing ambiguity while preserving cross-depth interaction.Source: §3.5 · pp. 5–6

3.6 DISCUSSION

  • Unlike U-ViT and U-DiTs, which fuse paired features through static learned projections, this method uses routing weights that adapt across inputs, positions, sublayers, and diffusion timesteps.It also differs from dense LLM residual routing by restricting sources to encoder-side representations and retrieving a single mirrored source per decoder layer.Source: §3.6 · p. 6

4 EXPERIMENTS

Across DiT scales and REPA-based fine-tuning, structured routing improves generation results with little parameter overhead. The analyses and ablations associate those gains with stage-matched, adaptive pathways that reshape feature and gradient structure.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers class-conditional ImageNet 256 × 256 generation, with FID, sFID, IS, Precision, and Recall reported without classifier-free guidance unless specified otherwise.Experiments include DiT-S/2, DiT-B/2, DiT-XL/2, and SiT-XL/2 + REPA, with comparisons to static and dense-routing variants.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 MAIN RESULTS

  • At 400K, FID falls from 69.81 to 62.48 (DiT-S/2), 44.21 to 36.33 (DiT-B/2), and 18.85 to 15.07 (DiT-XL/2), with added parameters below 0.1%.Across matched-FID comparisons, routing requires up to 1.73× fewer training steps.Source: §4.2 · p. 7, Table 1 · p. 7, Figure 5 · p. 8
  • After 0.35M routed fine-tuning iterations, REPA improves from 5.9 to 4.34 FID; with the guidance interval, the 0.28M model reaches 1.39 FID.Continued fine-tuning without routing brought no meaningful gain after REPA reached 5.9 FID.Source: §4.2 · p. 7
  • Routing adds 0.019M parameters and 0.3% FLOPs on DiT-S/2, while matching dense AttnRes at 62.48 vs. 62.87 FID and reducing source-stack activation by about 81%.The activation reduction is from 1.51 GB to 289 MB.Source: §4.2 · pp. 7–8

4.3 ANALYSIS: REPRESENTATION AND GRADIENT STRUCTURE

  • On DiT-XL/2, routing reduces excessive feature similarity and makes depth-wise structure clearer, with stronger early–late mirror-pair gradient alignment and differentiated middle layers.The mirror-pair gradient comparison uses layers i and 27 − i.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 ABLATION STUDIES

  • Among source-selection variants, mirror routing achieves the best FID and IS; adaptive fusion also outperforms a static projection when both connect the same mirror pairs.Dense routing over encoder-side sources attains a similar FID to AttnRes with a smaller source pool.Source: §4.4 · p. 9, Table 3 · p. 9

5 CONCLUSION

The conclusion frames residual connectivity in DiTs as adaptive source routing, with structured mirror routing improving representation and gradient structure and yielding gains across model scales and fine-tuning regimes.

5 CONCLUSION

  • Structured routing sends differentiable encoder-side representations to stage-matched decoder layers, improving depth-wise structure and results while matching dense AttnRes at lower activation cost.The experiments cover ImageNet and REPA-based fine-tuning, with negligible parameter overhead.Source: §5 · p. 9

A ADDITIONAL EXPERIMENTAL DETAILS

The additional experimental details specify the ImageNet setting, training protocols, and budgets used for DiT and REPA-based experiments.

A ADDITIONAL EXPERIMENTAL DETAILS

  • Experiments use class-conditional ImageNet 256 × 256 generation; DiT follows its original training protocol, while REPA fine-tuning adds routing modules without the projection loss for 0.28M or 0.35M steps.Unless specified otherwise, results are without classifier-free guidance; experiments use NVIDIA H200 GPUs.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 12

B ADDITIONAL QUANTITATIVE RESULTS

The appendix identifies additional routing-overhead and DiT-S/2 training-trajectory comparisons.

B ADDITIONAL QUANTITATIVE RESULTS

C ADDITIONAL ANALYSIS

Mirror-pair layers show increased gradient alignment, while middle layers remain more differentiated. The authors interpret this pattern as coordination in fine-detail encoding and reconstruction alongside semantic abstraction in middle layers.

C ADDITIONAL ANALYSIS

  • Early–late mirror pairs show increased gradient similarity, suggesting stage-matched pathways coordinate fine-grained feature encoding and reconstruction.The passage relates both early and late layers to dense visual details and high-frequency reconstruction.Source: C ADDITIONAL ANALYSIS · p. 12
  • Middle layers show lower gradient similarity, indicating functional differentiation that may support semantic abstraction and transformation.Source: C ADDITIONAL ANALYSIS · p. 12
  • At 400K iterations, mirror-pair similarity is measured by cosine similarity between gradients for layer i and layer 27 − i under a fixed probe loss.The probe uses the squared output norm on four random latents at t=500.Source: C ADDITIONAL ANALYSIS · p. 12

D ADDITIONAL VISUALIZATION RESULTS

Additional ImageNet 256 × 256 samples use REPA-XL/2 fine-tuned with structured residual routing and classifier-free guidance at w = 4.0. The authors describe the samples as visually faithful and high-quality across diverse categories.

D ADDITIONAL VISUALIZATION RESULTS

E LIMITATIONS

The reported improvements cover DiT-S/2, DiT-B/2, DiT-XL/2, and REPA-based fine-tuning, but broader validation remains future work. The authors identify larger multimodal models and video diffusion transformers as open settings, where conditioning or temporal dependencies may affect routing choices.

E LIMITATIONS

  • Broader validation on larger text-to-image and text-to-video models remains future work despite improvements on DiT-S/2, DiT-B/2, DiT-XL/2, and REPA-based fine-tuning.Source: E LIMITATIONS · p. 13
  • More complex conditioning, larger datasets, and different training recipes in these models may affect the optimal routing structure.Source: E LIMITATIONS · p. 13
  • It remains unclear whether mirror routing is always best under stronger multimodal conditioning or temporal dependencies.Source: E LIMITATIONS · p. 13

F BROADER IMPACT

The work aims to improve diffusion-transformer training efficiency and generation quality, with a stated positive impact on scalable visual generation. The authors also note potential misuse risks from image-generation advances and encourage responsible deployment practices.

F BROADER IMPACT

  • The work aims to improve diffusion-transformer training efficiency and generation quality through better residual connectivity.Source: F BROADER IMPACT · p. 13
  • The stated positive impact is enhancing high-quality generative modeling of scalable visual generation systems.Source: F BROADER IMPACT · p. 13
  • The authors note that improved image generation may increase misuse risks and encourage synthetic-content disclosure, careful dataset curation, and deployment safeguards.They state that the work introduces no new dataset or high-risk pretrained generative model.Source: F BROADER IMPACT · p. 13

Related papers

Loading 2609.33203v1…