PaperlayerView exact revision on arXiv

Source-linked AI summary

Should We Skip Diffusion?

Yiping Ji, James Martens, Simon Lucey

arXiv:2610.07002v1cs.LGcs.CV

TL;DR

DDT’s residual connections may make it harder to develop abstract encoder features while retaining information from multiple depths, even though the decoder needs both abstract structure and low-level detail. DDT-RFE removes those shortcuts and aggregates features at the encoder output; it improves over DDT across four visual tasks with fewer encoder blocks and achieves lower ImageNet FID.

  • Problem

    DDT’s residual connections carry earlier features into later transformations, and the paper asks whether the encoder can develop more abstract features while retaining multiple abstraction levels for the decoder.

    Source: §1 · p. 1

  • Method

    DDT-RFE removes residual connections around encoder Self-Attention and MLP operations and fuses input patch embeddings with intermediate and final encoder features for the decoder.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    DDT-RFE improves over DDT across four visual understanding tasks with fewer encoder blocks and achieves lower ImageNet FID under the same training budget.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The residual-free encoder and output feature aggregation provide more abstract representations while keeping features from different encoder depths accessible to the decoder.

    Source: §4 · p. 5, §7 · p. 9

    Experiments focus on DDT models trained on ImageNet at 256 × 256 resolution for 80 epochs, leaving longer schedules, higher resolutions, and other diffusion architectures for future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity decoder in denoising. To enable effective denoising at all noise levels, these features must capture both high-level abstract structures and low-level details. However, skip/residual connections in the encoder allow shallow features to bypass successive transformations, which may limit progressive abstraction, or at least make it difficult to disentangle different levels of abstraction. We propose DDT-RFE, which removes the residual connections around the Self-Attention and MLP operations in each encoder block while maintaining stable training. To retain the information that abstraction discards but that the decoder still needs, we fuse the input patch embedding with intermediate and final encoder features to form the encoder output. The decoder thus has access to information from multiple encoder depths, while each encoder block is able to learn more abstract representations. DDT-RFE achieves overall improvements over DDT across visual understanding tasks, including image classification, semantic segmentation, object discovery, and semantic correspondence, while using fewer encoder blocks. It also achieves a lower FID for image generation on ImageNet.

1 INTRODUCTION

DDT separates representation learning from signal reconstruction, but its residual connections may make it harder to develop abstract encoder features while retaining information from multiple depths. DDT-RFE removes those encoder shortcuts and fuses features from different depths, improving visual-task performance and ImageNet generation results.

1 INTRODUCTION

  • DDT uses a condition encoder for representations and a velocity decoder for reconstruction, while its encoder features must support both abstract structures and low-level image details.The paper hypothesizes that repeated accumulation of shallow features through residual connections may hinder abstraction.Source: §1 · p. 1
  • At timestep t = 0.5, PCA-projected encoder features are visualized as RGB images, and DDT-RFE shows more abstract representations with cleaner backgrounds and salient objects.The figure compares representations from shallow to deep layers.Source: Figure 1 · p. 2
  • DDT-RFE removes residual connections around Self-Attention and MLP operations, then fuses input patch embeddings and intermediate and final encoder features for the decoder.A proposed initialization scheme is intended to improve convergence of the residual-free encoder.Source: §1 · p. 2
  • DDT-RFE improves over DDT on four visual tasks with fewer encoder blocks and lowers ImageNet FID from 8.98 to 7.68 at L scale and from 7.27 to 6.81 at XL scale.The FID comparisons use the same training budget.Source: §1 · p. 2

2 RELATED WORK

Prior work develops diffusion-transformer architectures, studies semantic representations learned through diffusion, and examines how residual pathways affect optimization and feature persistence. This work focuses on abstraction in DDT’s condition encoder and on retaining information useful for reconstruction.

2 RELATED WORK

  • DiT (Peebles & Xie, 2023) provides a scalable latent-diffusion backbone, while U-DiT (Tian et al., 2024) adds long skip connections that carry shallow features to deeper layers.PixelDiT (Yu et al., 2026) separates patch-level semantic modeling from pixel-level detail refinement, and DDT (Wang et al., 2026) uses a condition encoder and velocity decoder.Source: §2 · pp. 2–3
  • Diffusion models have been studied as self-supervised representation learners, with prior work reporting linearly separable intermediate features, semantic correspondence, and noise-free semantic features.Examples include Xiang et al. (2023), DiTF (Gan et al., 2025), and CleanDIFT (Stracke et al., 2025).Source: §2 · p. 3
  • Prior studies link residual pathways both to improved network conditioning and to persistence of earlier-block information in deeper representations.Ji et al. (2025) discuss conditioning; Raghu et al. (2021) and Zhang et al. (2026) discuss information persistence.Source: §2 · p. 3
  • This work examines progressive abstraction in DDT’s condition encoder and aggregates features from different depths at its output for the decoder.Prior work reports that residual-free vision transformers can learn more abstract representations and improve dense prediction (Ji et al., 2026a).Source: §2 · p. 3

3 PRELIMINARIES

DDT uses a condition encoder to produce representations for a velocity decoder, operating on noisy latents. Its condition encoder forms patch tokens, applies conditioned transformer blocks, and—through residual accumulation—retains shallow features in deeper representations.

3.1 DECOUPLED DIFFUSION TRANSFORMER

  • DDT separates semantic extraction into a condition encoder and signal reconstruction into a velocity decoder, with both operating in a pretrained VAE’s latent space.The VAE encoder maps an image to a vectorized latent representation.Source: §3.1 · p. 3
  • The encoder maps noisy latent x_t to a representation under timestep and class conditioning, which is combined with the timestep embedding to condition the decoder.The decoder takes x_t as its input; its encoder output Z_t has N tokens of dimension D.Source: §3.1 · p. 3

3.2 CONDITION ENCODER ARCHITECTURE

  • The noisy latent is divided into non-overlapping patches, flattened, and linearly projected into D-dimensional tokens forming H_0.H_ℓ denotes the output of encoder block ℓ.Source: §3.2 · p. 4
  • Timestep and class information enter through adaptive layer normalization, with per-block affine layers producing shift, scale, and gate vectors for attention and MLP.The timestep and class embedding functions are e_t(·) and e_y(·), respectively.Source: §3.2 · p. 4
  • Each residual encoder block applies Self-Attention and MLP transformations with channel-wise modulation vectors broadcast across tokens.The passage also identifies learnable MLP weight matrices and the hidden dimension D_ff.Source: §3.2 · p. 4
  • Expanding the residual updates shows that shallow features persist through deeper blocks, and the final encoder output Z_t = H_L is supplied to the decoder.The expansion expresses the block outputs as accumulated residual updates.Source: §3.2 · p. 4

4 METHOD

DDT-RFE removes identity shortcuts within encoder blocks while aggregating features from multiple depths at the encoder output. Its initialization strategies address stable training of the residual-free encoder.

4 METHOD

  • DDT-RFE removes residual connections around attention and MLP sublayers while retaining timestep- and class-dependent modulation, encouraging progressive semantic abstraction.The blocks apply gated transformations without additive identity pathways.Source: §4 · p. 4
  • The encoder output aggregates input patch embeddings and every block output, giving the decoder direct access to features from different depths without internal identity pathways.This avoids requiring deeper blocks to preserve all information needed for reconstruction.Source: §4 · p. 5
  • Mimetic initialization sets query–key projections to encourage diagonal dominance, which can improve self-attention conditioning and support stable, fast convergence.The described initialization uses scalar constants α and β and an identity-matrix component.Source: §4 · p. 5
  • The proposed query/key initialization mixes a shared orthogonal component with independently sampled defaults, avoiding the need to tune α and β.The mixing parameter ρ controls the random components; at ρ = 0, both projections start from the same orthogonal matrix, while at ρ = 1 they retain default initialization.Source: §4 · p. 5
  • The MLP blocks use scale-corrected uniform orthogonal initialization to preserve input vector RMS while keeping the weight matrices' condition number at 1.For the trained depths, transforming the activation functions was unnecessary.Source: §4 · p. 5

5 EXPERIMENTS

The experiments evaluate diffusion-transformer representations across four visual tasks and class-conditional ImageNet generation, using models trained from scratch for 80 epochs. DDT-RFE improves results across the reported representation evaluations and lowers FID, with dataset-specific qualifications for segmentation.

5.1.1 IMAGE CLASSIFICATION

  • Despite fewer encoder blocks, DDT-RFE-L improves kNN accuracy over DDT-L by 9.8% at t = 0.1 and 6.4% at t = 0.3, and leads in linear-probe accuracy at both scales.Its best linear-probe performance occurs at t = 0.05, versus t = 0.15 for DDT; the encoder output also stays closer to its best intermediate representation.Source: Figure 2 · p. 6, §5.1.1 · pp. 6–7, Table 1 · p. 7

5.1.2 OBJECT DISCOVERY

  • DDT-RFE improves CorLoc over DDT by about 16% on VOC2012 and 10% on COCO20k at both scales.The authors link the results to more similar features for tokens belonging to the same object, which can help distinguish objects from background.Source: Table 2 · p. 7, §5.1.2 · p. 7

5.1.3 SEMANTIC CORRESPONDENCE

  • DDT-RFE improves PCK over DDT by 3.9 and 4.7 percentage points at L and XL, respectively, using 12 versus 20 and 14 versus 22 encoder blocks.These zero-shot gains come from directly matching frozen features without a task-specific prediction head.Source: Table 3 · p. 8, §5.1.3 · p. 8

5.1.4 SEMANTIC SEGMENTATION

  • On VOC2012, DDT-RFE gains about 6 mIoU at both scales; single-block scene-task gains are smaller, but multiscale probing makes it lead on every dataset and scale.At XL, single-block DDT-RFE matches DDT on COCO-Stuff and is 0.3 mIoU lower on ADE20K; multiscale probing benefits DDT-RFE more on both scene datasets.Source: Table 4 · p. 8, §5.1.4 · p. 8

5.2 IMAGE GENERATION

  • FID is lower for DDT-RFE at both scales: 7.68 vs 8.98 at L and 6.81 vs 7.27 at XL, under the same training budget.At XL, it also improves sFID and precision, matches IS, and has slightly lower recall.Source: Table 5 · p. 9, §5.2 · p. 9

6 LIMITATIONS AND FUTURE WORK

The experiments are limited to DDT models trained on ImageNet at 256 × 256 resolution for 80 epochs; broader settings remain future work.

6 LIMITATIONS AND FUTURE WORK

  • Experiments focus on DDT models trained on ImageNet at 256 × 256 resolution for 80 epochs.The authors cite limited computational resources as the reason for this scope.Source: §6 · p. 9
  • Longer training schedules are left for future experiments.Source: §6 · p. 9
  • Experiments at higher resolutions, such as 512 × 512, and with other diffusion architectures are left to future work to assess broader applicability.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents DDT-RFE as a residual-free condition encoder with feature aggregation at its output. It reports gains across four visual understanding tasks with fewer blocks and lower ImageNet FID under the same training budget.

7 CONCLUSION

  • DDT-RFE combines a residual-free encoder with output feature aggregation, allowing blocks to transform features without identity shortcuts while keeping multiple encoder depths accessible to the decoder.Source: §7 · p. 9
  • DDT-RFE improves over DDT across four visual understanding tasks using fewer encoder blocks and achieves lower ImageNet FID under the same training budget.Source: §7 · p. 9
  • Ablations show the importance of both the proposed initialization and output feature aggregation.Source: §7 · p. 9

A.1 GENERATION IMAGE

Figure 3 shows DDT-RFE-L/2 images generated on ImageNet at 256 × 256 resolution. The figure uses a 250-step Euler ODE sampler with cfg=4.

A.1 GENERATION IMAGE

  • Figure 3 shows images generated by DDT-RFE-L/2 on ImageNet at 256 × 256 resolution.Source: §A.1 · p. 12
  • The Figure 3 sampling setup uses a 250-step Euler ODE sampler.Source: Figure 3 · p. 12
  • The sampler uses cfg=4.Source: Figure 3 · p. 12

A.2 SELF-ATTENTION AND MLP OPERATIONS

The appendix describes the encoder-block Self-Attention and MLP operations, including their inputs and selected implementation details. It specifies that the MLP uses two layers and GELU independently on each token.

A.2 SELF-ATTENTION AND MLP OPERATIONS

  • In each encoder block, Self-Attention takes Hℓ−1 as input and the MLP takes H̃ℓ, with normalization and adaptive modulation defined in Eq. (6).Source: §A.2 · p. 12
  • The appendix omits normalization layers, rotary positional embeddings, and bias terms when detailing the operations.Source: §A.2 · p. 12
  • For Self-Attention, head outputs are concatenated and projected to form the operation's output.Source: §A.2 · p. 12
  • The MLP has two layers with a GELU activation and is applied independently to each token.Source: §A.2 · p. 12

A.3 ABLATION ON DDT

The ablations show that initialization and feature fusion matter to DDT-RFE’s performance, while additional PCA figures visualize representations across models and noise levels.

A.3 ABLATION ON DDT

  • Applying the proposed initialization and linear feature fusion to DDT degrades generation, yields comparable classification, and performs worse on semantic segmentation and object discovery.Tables 6 and 7 report the generation/classification and segmentation/object-discovery comparisons, respectively.Source: §A.3 · p. 13, Table 6 · p. 13, Table 7 · p. 13

A.4 ABLATION ON DDT-RFE

  • Removing the proposed initialization raises FID from 7.68 to 9.86 and substantially reduces visual-task performance; removing feature fusion causes still larger drops across generation and visual understanding.The initialization ablation’s visualizations show most representations collapsing, attributed to network conditioning issues.Source: §A.4 · p. 13, Table 8 · p. 13, Figure 4 · p. 14
  • The output-gate-removed variant still underperforms the full recipe, increasing FID by 0.66 and reducing classification accuracy, segmentation mIoU, and correspondence PCK by 0.2 percentage points, 1.1 points, and 4.4 percentage points, respectively.Its middle blocks learn some abstraction but still attempt to retain details because no alternative pathways are available.Source: §A.4 · p. 13

A.5 MORE VISUALIZATION

  • Figures 5–7 visualize representations from DDT-L/2 (20en4de), DDT-L/2 (12en12de), and DDT-RFE-L/2 (12en12de) at noise levels t = 0.1, 0.3, and 0.5.Figure 5 compares DDT and DDT-RFE at t = 0.1; Figures 6 and 7 show DDT-RFE at t = 0.3 and t = 0.5.Source: §A.5 · p. 15, Figure 5 · p. 15, Figure 6 · p. 16

Related papers

Loading 2610.07002v1…