PaperlayerView exact revision on arXiv

Source-linked AI summary

Persistence Forcing: Exploiting Feature Specialization in Pixel-Space Diffusion

Chong Wang, Zixuan Fu, Shiqi Huang, Siyuan Yang, Hao Cheng, Bihan Wen

arXiv:2609.36014v1cs.CV

TL;DR

Pixel-space DiTs typically refine all hidden features uniformly, although images contain both compact global structure and fine detail. The paper introduces heterogeneous refinement and Persistence Forcing to make persistent features guide active refinement and provide sampling-time guidance. PerF improves corresponding JiT results across reported scales and resolutions, including FID of 1.91 for PerF-L on ImageNet 256 × 256 and 1.76 for PerF-H/32 on 512 × 512.

  • Problem

    Pixel-space DiTs operate on high-dimensional visual data without pretrained latent representations, yet typically refine all hidden features uniformly while needing to organize global structure and fine detail.

    Source: §1 · pp. 1–2

  • Method

    Heterogeneous refinement assigns feature groups distinct refinement budgets, and Persistence Forcing uses persistent features as context for active refinement and sampling-time guidance.

    Source: §1 · p. 2

  • Results

    PerF improves corresponding JiT results across reported scales and resolutions; FID falls from 2.36 to 1.91 for PerF-L at ImageNet 256 × 256 and from 1.94 to 1.76 for PerF-H/32 at 512 × 512.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Persistent features provide stable global context for active refinement, while Persistence Guidance complements classifier-free guidance with an internally established structural signal.

    Source: §1 · p. 2

Abstract

from arXiv · show

Pixel-space diffusion Transformers (DiTs) directly operate on high-dimensional visual data, yet their hidden representations typically undergo uniform refinement across depth. Natural images, however, are inherently organized at different levels of granularity. Global structure can often be represented compactly, whereas local textures and fine details require richer representations. Motivated by this, we introduce heterogeneous refinement in pixel-space DiTs, assigning different feature groups distinct refinement budgets across depth. Consequently, an ordered feature specialization emerges: sparsely refined features predominantly encode global visual structure, whereas more frequently refined features increasingly specialize toward localized, high-frequency details. We refer to these two groups as persistent and active features, respectively. Building on this emergent specialization, we introduce Persistence Forcing (PerF), which explicitly exploits this persistent--active feature organization for pixel-space image generation. This enables persistent features to continuously condition actively refined features, allowing stable global information to guide the ongoing refinement of finer visual details. During generative sampling, this interaction further induces a meaningful guidance direction that promotes coherent global structure and naturally complements classifier-free guidance. On ImageNet $256\times256$, PerF-L achieves FID of $1.91$, approaching $1.86$ of JiT-H with only half the parameters, while PerF-H further achieves FID of $1.63$ and $1.76$ on ImageNet $256\times256$ and $512\times512$, respectively.

1 INTRODUCTION

The introduction argues that pixel-space DiTs must organize global structure and fine detail despite typically refining all hidden features uniformly. It presents heterogeneous refinement and Persistence Forcing as a way to exploit persistent global features while actively refining detail.

1 INTRODUCTION

  • Pixel-space DiTs operate directly on high-dimensional visual data, yet typically refine the full hidden representation uniformly despite images combining global structure with fine detail.The authors identify learning effective internal representations as a key bottleneck in training generative DiTs.Source: §1 · pp. 1–2
  • Heterogeneous refinement yields persistent features that predominantly encode global structure and active features that increasingly specialize toward localized, high-frequency detail.The introduction describes these roles as emerging from different refinement budgets.Source: §1 · p. 2
  • PerF reduces FID from 3.66 to 2.81 for JiT-B, from 2.36 to 1.91 for JiT-L, and from 1.86 to 1.63 for JiT-H on ImageNet 256 × 256; PerF-H/32 reduces it from 1.94 to 1.76 at 512 × 512.The introduction also reports an Inception Score of 335.3 for PerF-H/32 at 512 × 512.Source: §1 · p. 2
  • Persistence Forcing uses persistent features as stable context for active refinement and applies sampling-time guidance that complements classifier-free guidance.Persistence Guidance contrasts predictions with and without persistent conditioning to strengthen the influence of internally established structural context.Source: §1 · p. 2

2 HETEROGENEOUS REFINEMENT INDUCES FEATURE SPECIALIZATION

The section introduces variable-width heterogeneous refinement and reports that it produces ordered feature specialization, but that refinement alone does not improve generation quality. This motivates explicitly using persistent representations to support ongoing active computation.

2.1 HETEROGENEOUS REFINEMENT VIA VARIABLE WIDTH

  • A contraction–expansion schedule varies the active width across depth, so feature groups receive different refinement budgets while bypassed features can re-enter later computation.The design approximately matches the attention and MLP parameter budget of the corresponding JiT backbone.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 EMERGENT FEATURE SPECIALIZATION

  • Different refinement histories produce an ordered coarse-to-fine progression: limited-refinement groups preserve global structure, while more-refined groups contribute localized textures, boundaries, and details.The paper calls sparsely refined global-structure features persistent features and repeatedly transformed features actively refined features; these roles emerge rather than being explicitly assigned.Source: §2.2 · p. 3

2.3 HETEROGENEOUS REFINEMENT ALONE IS NOT ENOUGH

  • Heterogeneous refinement alone does not improve generation quality: a parameter-matched model can have worse FID than vanilla JiT despite showing coarse-to-fine specialization.The authors motivate explicitly using persistent representations because the basic construction carries them forward without using them to support active computation.Source: §2.3 · p. 4

3 PERSISTENCE FORCING

Persistence Forcing preserves persistent features as structural context for actively refined features. It also uses that controllable interaction to derive sampling-time guidance toward coherent global structure.

3.1 PERSISTENT-TO-ACTIVE CONDITIONING

  • Persistent features remain preserved and provide spatially aligned, token-wise modulation that guides active refinement through a low-rank bottleneck.The persistent representation informs active features without itself being repeatedly transformed.Source: §3.1 · p. 4

3.2 PERSISTENCE GUIDANCE

  • Stochastically disabling persistent conditioning during training lets one model predict with and without that internal context, creating Persistence Guidance.The difference between these predictions provides an interpretable contrast for sampling.Source: §3.2 · p. 5
  • The resulting contrast supplies sampling-time guidance toward established global structure, with wp > 1 strengthening persistent-context influence.Setting wp = 1 recovers the standard persistent-conditioned prediction.Source: §3.2 · p. 5
  • Persistence Guidance complements CFG: CFG strengthens class-consistent appearance, while PG promotes coherent global organization.The two guidance signals arise from different conditioning sources.Source: §3.2 · p. 5

4 EXPERIMENTS

PerF improves JiT across ImageNet resolutions and model scales, then ablations show how heterogeneous refinement, persistent-to-active conditioning, and Persistence Guidance contribute. The experiments also test schedule, conditioning-capacity, and guidance choices.

4.1 IMPLEMENTATION DETAILS

  • The experiments use class-conditional ImageNet generation with PerF built on JiT architectures and a contraction–expansion width schedule.Persistent conditioning uses a low-rank bottleneck, with reported overhead below 5% across model scales.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • FID falls from 2.36 to 1.91 for PerF-L on ImageNet 256 × 256, and from 1.94 to 1.76 for PerF-H/32 on ImageNet 512×512 versus JiT-H/32.PerF-H reaches FID 1.63 on ImageNet 256 × 256; the gains are reported across model scales and resolutions.Source: Table 2 · p. 6, Table 3 · p. 6, §4.2 · p. 7

4.3 ABLATION STUDY

  • FID varies from 40.35 to 41.35 across four tested schedules, versus 42.1 for JiT, and the Late Bottleneck schedule is adopted by default.All tested schedules outperform JiT at the same training stage; Figure 5 presents the schedule ablation.Source: Figure 5 · p. 7, §4.3 · p. 7
  • Adding P-to-A conditioning reduces FID from 42.71 to 40.35 without CFG and from 8.02 to 7.19 with CFG; Persistence Guidance further reduces it to 24.92 and 5.81, respectively.PG uses the prediction contrast between enabling and disabling the P-to-A pathway; heterogeneous refinement alone does not improve generation quality.Source: §4.3 · pp. 7–8, Table 4 · p. 8
  • With CFG disabled, increasing the P-to-A bottleneck rank from 128 to 256 changes FID from 40.35 to 40.06 without PG while doubling added parameters from 5.7M to 11.5M.The default rank is 128 as a trade-off between quality and parameter overhead.Source: §4.3 · p. 8, Table 5 · p. 8
  • Moderate Persistence Guidance improves FID across tested CFG scales, while excessive PG strength degrades performance; applying PG over [0, 1] gives the best overall performance.The reported guidance comparison describes PG as reinforcing internal structural context while CFG emphasizes external semantic information.Source: §4.3 · pp. 8–9, Figure 6 · p. 8, Table 6 · p. 8

5 RELATED WORK

Related work motivates organizing pixel-space representations into distinct roles and improving intermediate representations. It also situates PerF among sampling-guidance methods that derive signals from model predictions or internal computation.

5.1 LEARNING REPRESENTATIONS IN PIXEL-SPACE DIFFUSION TRANSFORMERS

  • Prior pixel-space methods assign distinct roles through patch, spatial, or frequency organization, while representation-alignment methods target intermediate diffusion features.These approaches motivate examining roles that emerge from heterogeneous computation itself.Source: §5.1 · p. 9

5.2 GUIDANCE FOR DIFFUSION MODELS

  • Guidance methods contrast conditional predictions, model qualities, or perturbed computation paths; PerF instead amplifies a contrast with and without persistent structural context.The resulting signal reinforces internal structure and complements CFG's external semantic conditioning.Source: §5.2 · p. 9

6 CONCLUSION

The conclusion presents PerF as a way to induce and exploit persistent–active feature specialization, reporting gains over JiT at two ImageNet resolutions. The related-work discussion contrasts this emergent organization with methods that prescribe spatial, frequency, or computational decompositions.

6 CONCLUSION

  • PerF assigns different refinement budgets to feature groups, yielding persistent features for global structure and active features for finer, high-frequency details.The conclusion reports FIDs of 1.63 and 1.76 for PerF-H on ImageNet 256 × 256 and 512 × 512, respectively, improving on corresponding JiT baselines.Source: §6 · p. 9

A EXTENDED RELATED WORK

  • Unlike prior methods that prescribe spatial, frequency, or temporal organization, PerF uses heterogeneous refinement to make coarse-to-fine roles emerge across feature groups.The related-work discussion also distinguishes PerF from approaches focused primarily on computational efficiency or capacity allocation.Source: A EXTENDED RELATED WORK · p. 13
  • The depth-wise visualization shows persistent features acquiring global object structure after one update and remaining stable during bypass, while active features continue evolving across blocks.The persistent group is updated at Blocks 1 and 12; active features are updated at every block.Source: Figure 7 · p. 14

B.1 DEPTH-WISE EVOLUTION OF PERSISTENT AND ACTIVE FEATURES

Depth-wise analyses distinguish persistent features that preserve global organization from active features that continue to evolve. Frequency measurements further associate larger refinement budgets with finer spatial content across diffusion timesteps.

B.1 DEPTH-WISE EVOLUTION OF PERSISTENT AND ACTIVE FEATURES

  • Persistent features show coherent global structure after one update and remain stable through bypass, while active features evolve across depth with residual and high-frequency patterns before later local structures emerge.Persistent features are updated at Blocks 1 and 12; active features are updated throughout the network.Source: §B.1 · pp. 14–15

B.2 QUANTIFYING FEATURE SPECIALIZATION IN THE FREQUENCY DOMAIN

  • Feature groups with refinement budgets R = 2–6 are dominated by low spatial frequencies, while groups with R ≥8 contribute increasingly finer structure across evaluated timesteps.The overall frequency shift is not strictly monotonic at every individual timestep; Figure 8 reports the centroid comparison.Source: Figure 8 · p. 15, §B.2 · pp. 15–16
  • The normalized spectral centroid measures the energy-weighted mean radial spatial frequency of each feature group's RGB contribution, using equal-sized 64-dimension subsets.Higher centroid values indicate a greater concentration of contribution toward finer spatial frequencies.Source: §B.2 · p. 15
  • The frequency results support a coarse-to-fine organization in which sparsely refined features carry global structure and more refined features specialize toward finer details.The reported comparison uses equal-sized 64-channel subsets to control for the number of contributing channels.Source: §B.2 · pp. 15–16

C IMPLEMENTATION DETAILS

The main PerF models follow JiT’s optimization and noise-sampling setup, with a fixed 600-epoch training schedule. Sampling uses Heun with 50 steps and specified CFG and PG scales.

C IMPLEMENTATION DETAILS

  • PerF backbones approximately match corresponding JiT attention and MLP parameter budgets, with a small additional parameter overhead for persistent-to-active conditioning.PerF uses a contraction–expansion schedule; PerF-H/16 and PerF-H/32 share the same depth, dimensions, and width schedule.Source: C IMPLEMENTATION DETAILS · p. 17
  • All main models train for 600 epochs with Adam, batch size 1024, learning rate 2 × 10^-4, and no weight decay.The timestep follows logit(t) ∼ N(−0.8, 0.82); noise scale is 1.0 at 256 × 256 and 2.0 at 512 × 512.Source: C IMPLEMENTATION DETAILS · p. 17
  • Main scaling results use REPA only during the first 100 epochs; extending alignment throughout training degrades final generation quality.Retrained ablations omit REPA unless stated otherwise, while sampling-only PG analyses reuse main-experiment checkpoints.Source: C IMPLEMENTATION DETAILS · p. 17
  • All reported results use Heun with 50 sampling steps, applying CFG over [0.1, 1.0] and PG over [0, 1] unless otherwise specified.Model-specific CFG scales are 2.8, 2.7, 3.0, and 2.9, with corresponding PG scales of 3.0, 2.8, 1.9, and 2.0.Source: C IMPLEMENTATION DETAILS · p. 17

D IMPLEMENTATION OF PERSISTENT-TO-ACTIVE CONDITIONING

Persistent features are projected into modulation terms that condition active features, while persistent features bypass each block unchanged. PerF improves over JiT without CFG, and early-only REPA gives the strongest reported alignment schedule.

D IMPLEMENTATION OF PERSISTENT-TO-ACTIVE CONDITIONING

  • At each block, a learned low-rank projector maps normalized persistent features into six modulation components for active-feature attention and MLP sublayers.The components augment timestep- and class-conditioned AdaLN modulation; persistent features bypass the block and are concatenated back afterward.Source: D IMPLEMENTATION OF… · p. 17
  • The persistent-to-active pathway is independently dropped with probability 0.1 during training, enabling predictions with or without persistent conditioning for Persistence Guidance.Each projector’s final linear layer is zero-initialized, so its conditioning branch begins as a zero residual.Source: D IMPLEMENTATION OF… · p. 17

E GENERATION WITHOUT CLASS-FREE GUIDANCE

F INTERACTION WITH REPRESENTATION ALIGNMENT

G ARCHITECTURAL OVERHEAD

PerF adds limited parameter and single-forward computation overhead relative to corresponding JiT backbones across model scales.

G ARCHITECTURAL OVERHEAD

H ADDITIONAL QUALITATIVE RESULTS

The paper presents additional uncurated class-conditional samples generated by PerF-H/16 on ImageNet at 256 × 256.

H ADDITIONAL QUALITATIVE RESULTS

Related papers

Loading 2609.36014v1…