PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Representations to Guide Pixel Diffusion Transformers

Zihao Pan, Mengting Chen, Cheng Shi, Taihang Hu, Zhengrong Yue, Zikang Wang, Yuan Wang, Wei Li, Sibei Yang

arXiv:2610.12978v1cs.CV

TL;DR

The paper asks what makes internal representations useful for generation, where models must organize semantics and spatial structure while recovering every pixel. It introduces RG, which forms a compact representation and uses it to guide synthesis, and reports strong results across image and text-to-image tasks.

  • Problem

    Visual representations are developed predominantly for understanding, leaving unclear whether a good representation for recognition is also useful for generation, which must organize semantics and spatial structure while recovering every pixel.

    Source: §1 · p. 2

  • Method

    RG combines a low-dimensional bottleneck that forms a compact representation with a representation-induced velocity offset that modifies the flow-matching target.

    Source: §1 · p. 3

  • Results

    Across ImageNet resolutions and text-to-image generation, RG reports strong results, including FID 1.55 for RG-XL/16 on ImageNet 256 × 256.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Useful generative representations must be both properly formed and actively exploited.

    Source: §6 · p. 11

    Experiments focus on image generation; larger model scales, video, 3D, and audio remain future work.

    Source: §6 · p. 11

Abstract

from arXiv · show

Visual representations are largely developed for understanding, yet what makes them useful for generation remains unclear. We study this question in pixel diffusion transformers and uncover two key insights. First, better representation quality does not necessarily translate into greater generative utility, as externally aligned semantics can be progressively lost during pixel synthesis. Second, architectural separation with specialized pixel decoders mainly redistributes semantic and spatial modeling, without necessarily yielding representations better suited to generation. These insights suggest that a useful generative representation must be both properly formed and actively exploited. Motivated by these insights, we introduce Representation Guidance (RG), which directly forms and exploits internal representations while retaining a plain Transformer architecture. A low-dimensional bottleneck forces generation-relevant information into a compact representation, which is then explicitly used to guide decoder generation by modifying the flow-matching target according to its induced prediction offset. RG learns clearer semantic and spatial structure while scaling efficiently in both parameters and computation. On ImageNet 256x256, RG-L/16 with only 459M parameters outperforms substantially larger pixel diffusion transformers, while RG-XL/16 achieves an FID of 1.55. RG further reaches an FID of 1.74 at 512x512 and extends to text-to-image generation, achieving 0.89 on GenEval and 87.1 on DPG-Bench.

1 Introduction

The paper asks what makes a representation useful for generation, where models must preserve semantics and spatial structure while recovering every pixel. It introduces Representation Guidance (RG), which forms a compact internal representation and explicitly uses it to guide synthesis.

1 Introduction

  • Visual representations have been developed mainly for understanding, leaving unclear whether a good representation for recognition is also useful for generation.Generation must organize global semantics and spatial structure while recovering every pixel, unlike recognition, which can discard much low-level image content.Source: §1 · p. 2
  • Externally aligned semantic and spatial gains can fade during pixel synthesis, and alignment can accelerate early training without improving final generation.Deeper alignment further degrades final FID, while specialized pixel decoders redistribute semantic and spatial modeling without necessarily producing representations better suited to generation.Source: §1 · pp. 2–3
  • A useful generative representation must be both properly formed and actively exploited by subsequent synthesis.The paper argues that compactness encourages generation-relevant organization, while explicit use keeps the representation coupled to synthesis.Source: §1 · p. 3
  • RG combines a low-dimensional bottleneck that forms a compact representation with a representation-induced velocity offset that modifies the flow-matching target.The bottleneck shapes how the representation is formed, while guidance encourages generation to use it; gradients through the representation also shape the encoder for synthesis.Source: §1 · p. 3
  • RG-XL/16 achieves an FID of 1.55 on ImageNet 256 × 256, while RG-L/16 reaches 1.79 with 459M parameters and outperforms substantially larger pixel-space models.RG also achieves an FID of 1.74 at 512 × 512 and extends to text-to-image generation, reaching 0.89 on GenEval and 87.1 on DPG-Bench.Source: §1 · pp. 3–4

2 What Makes Representations Useful in pDiTs?

The analysis finds that stronger intermediate representations do not necessarily improve generation, and that specialized decoders mainly redistribute semantic and spatial modeling. Decoder choice reshapes what the encoder retains, while neither alignment nor architectural separation alone ensures a better generative representation.

Experimental setup

  • The experiments examine whether representation quality survives synthesis, using FID-50K for generation, RMSC and LDS for spatial structure, and linear probing for semantics.Unless otherwise specified, models are base-sized and trained for 200 epochs on ImageNet 256 × 256.Source: §2 · p. 4

Representation alignment

  • Baseline finishes at FID 4.56 versus 4.80 for REPA-4; deeper REPA-6 and REPA-8 reach FIDs of 5.38 and 8.33.REPA-4 accelerates early training, but its representation gains diminish after the aligned layer and the baseline overtakes it after roughly 160 epochs.Source: §2 · p. 4
  • Layer-4 alignment improves RMSC, LDS, and linear probing accuracy, but these gains diminish immediately as the network approaches pixel prediction.The analysis suggests that later layers overwrite externally imposed semantics as they specialize toward dense pixel interactions and high-frequency synthesis.Source: §2 · p. 4

Architectural separation

  • Vanilla and PixelDiT retain stronger global semantics, whereas DiP and DeCo preserve more spatial structure but weaker semantics.Vanilla and PixelDiT model cross-patch spatial relations in the decoder; DeCo and DiP process patches largely independently, so their encoders retain more spatial information.Source: Figure 5 · p. 5, §2 · p. 5
  • PixelDiT roughly matches vanilla (4.26 vs. 4.17) with 55% more parameters and 67% more FLOPs, while DiP and DeCo reach FIDs of 6.09 and 9.17.Specialized decoders produce different semantic–spatial balances rather than a consistent generation gain.Source: Table 1 · p. 5, §2 · p. 5

3 Method

The method uses pixel-space flow matching, then adds a compact internal representation and trains the decoder to use it through representation guidance. The training setup combines this guidance with alignment supervision and applies it only under specified conditions.

3.1 Preliminaries

  • Pixel-space flow matching interpolates clean images and Gaussian noise, with the target velocity given by their difference.The network predicts the clean image, which is converted to a velocity.Source: §3.1 · p. 5

3.2 Representation Guidance

  • A low-dimensional bottleneck forms an implicit latent space that is computed from each noisy input while diffusion remains in pixel space.The encoder's representation conditions the decoder's clean-image prediction.Source: §3.2 · p. 6
  • Representation guidance shifts the flow-matching target along the offset between decoder predictions with and without the representation, using a stop-gradient offset.The RG loss reduces to the plain flow-matching loss when s = 1; gradients also shape the representation through the guided objective.Source: §3.2 · pp. 6–7

3.3 Training and Sampling

  • Training applies EMA-teacher guidance only above a noise-time threshold, combines RG with REPA alignment, and adds no inference cost.The guidance interval excludes highly noisy inputs, whose representations are described as unreliable; training also uses a plain flow-matching warm-up.Source: §3.3 · p. 7

4 Experiments

RG achieves strong class-conditional ImageNet and text-to-image results, while its internal features show semantic structure in encoder layers and high-frequency patterns in decoder layers. Ablations examine the representation bottleneck, alignment, architecture split, guidance settings, and alignment teacher.

4.1 Implementation Details

  • Experiments evaluate Base, Large, and Extra Large models on class-conditional ImageNet and text-to-image generation, using RG-B/16 for most ImageNet experiments.Class-conditional evaluations use ImageNet-1K at 256 × 256 and 512 × 512; text-to-image models are evaluated at 512 × 512.Source: §4 · p. 7, §4.1 · p. 8

4.2 Main Results

  • FID 1.55 on ImageNet 256 × 256 makes RG-XL/16 the lowest among pixel-space methods; RG-L/16 reaches FID 1.79 with 459M parameters, and RG-XL/16 reaches FID 1.74 at 512 × 512.RG-L/16 also outperforms JiT-G/16 (2B, FID 1.82) and PixelREPA-H/16 (953M, FID 1.81).Source: §4.2 · p. 8, Table 3 · p. 8
  • RG-T2I-L/16 achieves 0.89 on GenEval and 87.1 on DPG-Bench at 512 × 512, outperforming the cited large-scale latent-space and pixel-space methods.With the same model size and training recipe as MiniT2I-L/16, it improves GenEval by 0.01 and DPG-Bench by 1.2.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Representation Analysis

  • RG maintains clear semantic contours and foreground–background separation in encoder layers 3–8, while high-frequency patterns dominate decoder layers 9–12.The bottleneck improves spatial structure at encoder layer 8; guidance further strengthens functional specialization and global semantics.Source: §4.3 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9

4.4 Ablations

  • At 600 epochs, bottleneck width d = 128 achieves FID 2.64, below d = 512 at FID 2.89; at 200 epochs, tested widths yield FIDs from 3.18 to 3.36.The authors use d = 128 by default.Source: §4.4 · p. 10, Table 5 · p. 10
  • Aligning the last encoder layer raises FID to 8.97; an 8/4 encoder–decoder split yields FID 3.28, while a 10/2 split yields 4.19.Aligning layers 3 and 6 yields similar FIDs of 3.28 and 3.33; increasing encoder depth from a 4/8 to an 8/4 split reduces FID from 3.60 to 3.28.Source: §4.4 · p. 10, Table 6 · p. 10, Table 7 · p. 10
  • Adding the bottleneck and then guidance reduces FID to 3.79 and 3.28, respectively; without either REPA or the bottleneck, guidance raises FID to 7.85–8.99.REPA alone yields FID 4.17, and RG adds 0.4M parameters and 13% FLOPs.Source: §4.4 · p. 10, Table 8 · p. 10
  • RG scale s = 1.4 and interval lower bound tmin = 0.3 yield the lowest reported FIDs; applying guidance at all noise levels raises FID to 5.28.At 200 epochs, DINOv2-B, DINOv2-B with registers, and DINOv3-B/16 yield FIDs of 3.46, 3.28, and 3.30, respectively.Source: §4.4 · pp. 10–11, Figure 9 · p. 10

5 Related Work

Related work covers pixel-space generation, representation learning for generation, and diffusion guidance. The paper positions RG as learning internal representations end-to-end and modifying the training target with those representations.

Pixel-Space Generation

  • Pixel-space generation models RGB images directly, and recent Transformer approaches include patch-level modeling, pixel refinement, frequency decomposition, and encoder–decoder architectures.The paper identifies joint semantic representation learning and dense pixel synthesis from raw inputs as a convergence challenge.Source: §5 · p. 11

Representation Learning for Generation

  • Prior generation methods use pretrained visual representations as diffusion latents or align intermediate features to frozen encoders, while dedicated encoder–decoder designs separate semantic modeling from pixel synthesis.The paper contrasts these approaches with end-to-end representation learning under the generative objective, using a bottleneck and representation guidance.Source: §5 · p. 11

Guidance in Diffusion Models

  • Unlike guidance methods focused on sampling, RG modifies the flow-matching target during training using the model's own representation, with gradients shaping the vector field and internal features.The section contrasts RG with classifier guidance, classifier-free guidance, Internal Guidance, and RAEv2 self-guidance.Source: §5 · p. 11

6 Conclusion

Representation Guidance guides pixel diffusion transformers with learned representations, achieving FIDs of 1.55 and 1.74 on ImageNet at 256 × 256 and 512 × 512, respectively. The experiments focus on image generation; extension to larger scales and other modalities remains future work.

6 Conclusion

  • RG achieves FIDs of 1.55 and 1.74 on ImageNet 256 × 256 and 512 × 512, respectively, and extends to text-to-image generation.The paper concludes that useful generative representations must be both properly formed and actively exploited.Source: §6 · p. 11
  • Experiments focus on image generation; larger model scales, video generation, 3D, and audio are left for future work.Source: §6 · p. 11

A.1 Model Configurations

The appendix points to a table listing the architecture, training, and inference configurations of all RG models on ImageNet.

A.1 Model Configurations

  • Table 9 lists architecture, training, and inference configurations for all RG models on ImageNet.Source: §A.1 · p. 16

A.2 Analysis of Representation Guidance

Representation Guidance replaces the flow-matching target with a representation-guided target, shifting guidance from sampling into training. Its objective both induces a CFG-style velocity and trains the encoder representation to carry useful denoising information.

A.2 Analysis of Representation Guidance

  • RG replaces the flow-matching target with vRG, moving guidance into the training objective rather than sampling-time extrapolation.Sampling-time extrapolation along Δvrep would double decoder cost and leave the representation unchanged.Source: §A.2 · p. 16
  • At convergence, assuming the EMA teacher matches the online network, the representation-free branch converges to the plain conditional expectation ¯v∅.The minimizer of the objective is derived under this assumption.Source: §A.2 · p. 16
  • RG trains a CFG-style velocity with effective scale wrep = 1/(2−s), using the learned representation as its condition.For s = 1.4, wrep ≈ 1.67; a larger s reduces the CFG scale needed at sampling time.Source: §A.2 · p. 16
  • RG's shifted target makes the representation-conditioned branch deviate from the representation-free branch, requiring r to carry more denoising information.The LRG gradient flows through r into the bottleneck and encoder, shaping the representation for generation.Source: §A.2 · p. 16

A.3 Relation to Autoguidance and Internal Guidance

RG differs from autoguidance and Internal Guidance by isolating the contribution of a compact learned representation and shaping that representation through training. In pixel-space experiments, IG worsens JiT-B generation, while RG improves FID and its guidance is most effective when the representation is well formed and applied selectively.

A.3 Relation to Autoguidance and Internal Guidance

  • Autoguidance uses a separately trained weaker model, IG an intermediate auxiliary head, and RG the full model versus its representation-free sub-network.Figure 10 contrasts CFG guidance at sampling along Δvy with RG guidance during training along Δvrep.Source: §A.3 · p. 16, Figure 10 · p. 17
  • Unlike autoguidance and IG, RG's guidance direction isolates information carried by r, and its training gradient shapes the encoder representation.RG requires neither a second model nor an auxiliary head and adds no inference cost.Source: §A.3 · p. 16
  • IG's block-4 auxiliary head collapses spatial structure while leaving global semantics unchanged.At block 4, RMSC falls from 0.46 to 0.28, LDS from 0.11 to 0.04, and probing accuracy from 21.2% at layer 3 to 19.1%.Source: Figure 11 · p. 17, §A.3 · p. 17
  • JiT-B with IG raises FID from 4.56 to 5.22, while RG lowers FID from 3.79 to 3.28 under the reported 200-epoch ImageNet 256 × 256 comparison.Both methods lower the optimal CFG scale, but the IG change does not improve generation.Source: §A.3 · p. 17, Table 10 · p. 19
  • IG's shallow pixel prediction is nearly as accurate as the final output, with flow-matching loss 0.0696 versus 0.0667 and prediction-difference RMS 0.107.The passage conjectures that the remaining residual mainly contains fine pixel detail rather than semantics or layout.Source: §A.3 · p. 17
  • RG forms r at a compact bottleneck without asking it to reproduce pixels, while the decoder handles pixel synthesis from zt.The RGB encoder retains high spatial diversity and locality, with RMSC up to 0.91 and LDS up to 0.66.Source: §A.3 · p. 17
  • RG's Δvrep removes only r, isolating decoder reliance on the representation; guidance helps only with a well-formed representation.Without REPA and the bottleneck, guidance raises FID to 8.08; with both, it lowers FID from 3.79 to 3.28, while applying it at all noise levels raises FID to 5.28.Source: §A.3 · p. 18

A.4 Representation Metrics

The section defines how representation metrics are computed and evaluated, covering spatial distinctiveness, local-versus-distant similarity, and linear-probe semantics.

A.4 Representation Metrics

  • Representation metrics use EMA weights at epoch 200; decoupled-model layers 1–8 are encoder blocks and 9–12 decoder blocks, each with 256 patch tokens of dimension 768.The evaluation excludes in-context class tokens.Source: §A.4 · p. 19
  • RMSC measures spatial distinctiveness: higher values mean tokens at different positions are more distinct.It is averaged over the 50K ImageNet validation images.Source: §A.4 · p. 19
  • LDS contrasts nearby and distant patch-token similarity; neighbors are horizontal or vertical, while distant pairs have Manhattan distance at least 6.Scores are averaged per image over the 50K validation images.Source: §A.4 · p. 19
  • Linear probing averages each block’s patch tokens into image-level features and reports ImageNet validation top-1 accuracy from a classifier trained on ImageNet training features.The classifier is preceded by a BatchNorm layer without affine parameters.Source: §A.4 · p. 19

A.5 Additional Results

Additional experiments examine RG’s training behavior, guidance settings, and layer-wise representations. Qualitative figures show uncurated ImageNet and text-to-image samples across resolutions.

A.5 Additional Results

  • A larger RG scale shifts the optimal CFG scale lower, while applying RG at all noise levels (tmin = 0) leaves FID high across CFG scales.The reported sweep evaluates B/16 models at epoch 200 over CFG scales in (0.1, 1].Source: §A.5 · p. 19, Figure 12 · p. 20
  • Enabling RG at epochs 0, 10, or 30 instead of the default epoch 20 changes FID by less than 0.01, making warm-up length a small factor in these experiments.The authors conjecture guidance grows as the representation becomes informative.Source: §A.5 · p. 20
  • The online guidance teacher lowers optimal FID from 3.28 to 3.18 and raises IS from 264.1 to 272.3 compared with the EMA teacher.Both results come from single runs; other paper results use the EMA teacher.Source: Figure 13 · p. 20, §A.5 · p. 20
  • RG-B matches JiT-B’s 600-epoch FID of 3.66 after 150 epochs with FID 3.64, a 4× speed-up, and reaches 2.64 at 600 epochs.REPA speeds up JiT-B early, but that advantage disappears after about 160 epochs.Source: Figure 14 · p. 21, §A.5 · p. 21
  • RG-B’s last encoder layer has RMSC 0.858 and LDS 0.390, versus JiT-B’s 0.476 and 0.111; RG-B’s FID is 3.28 versus REPA’s 4.80 despite lower peak accuracy.RG-B maintains semantic and spatial diversity through the encoder interface, while its decoder performs pixel synthesis.Source: Figure 15 · p. 21, §A.5 · p. 21

A.6 Qualitative Results

  • Uncurated examples show RG-XL/16 ImageNet samples at 256×256 and 512×512, plus text-to-image samples at 512×512 and 1024×1024.The 1024×1024 text-to-image examples use a model fine-tuned from the 512×512 model.Source: Figure 16 · p. 22, Figure 28 · p. 25, Figure 29 · p. 26

Related papers

Loading 2610.12978v1…