PaperlayerView exact revision on arXiv

Source-linked AI summary

A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples

Zixuan Fu, Chong Wang, Lanqing Guo, Kailai Zhou, Jiahao Nie, Bihan Wen

arXiv:2607.29122v1cs.CV

TL;DR

Pixel-space diffusion must model global structure and local texture directly in a high-dimensional pixel space, while existing improvements usually require retraining a model from scratch. SSG instead uses a frozen pretrained model’s intermediate-to-final refinement discrepancy for self-guidance, improving FID across multiple ImageNet pixel diffusion models with adapter training below 1% of full-model compute.

  • Problem

    Pixel-space diffusion must capture global semantics and fine-grained details in one high-dimensional space, while most improvements require expensive training of a new model from scratch.

  • Method

    SSG attaches a lightweight adapter to an intermediate frozen layer, trains it on model-generated samples, and uses its discrepancy from final predictions as sampling-time self-guidance.

  • Results

    More than 50% FID reduction occurs across evaluated JiT variants without CFG, while CFG results improve JiT-H/16 from 1.86 to 1.67 and PixelREPA-H/16 from 1.81 to 1.59.

  • Takeaways & Limitations

    SSG provides a low-cost plug-in for improving several pretrained pixel diffusion models while keeping their backbones frozen and using no real data for adapter training.

  • Takeaways & Limitations

    Experiments mainly cover class-conditional ImageNet generation, so generalization to larger text-conditioned models and domains with different image distributions remains unclear.

Abstract

from arXiv · show

Pixel-space diffusion models aim to learn an end-to-end generator directly over raw pixels. This is challenging because a single model must capture both global structure and local texture in the same high-dimensional space. While recent work improves pixel diffusion through alternative prediction targets, training objectives, and architectures, these advances typically require training a new model from scratch. We show there is a cheaper, complementary strategy: \textbf{a frozen, pretrained pixel diffusion model can guide itself}. Our key observation is that intermediate layers of a pretrained pixel diffusion transformer can be decoded into coarse predictions that capture the main low-frequency structure, while the final layers progressively refine local, high-frequency details. We therefore attach a lightweight prediction head to an intermediate layer, keep the backbone frozen, and use the discrepancy between the intermediate and final predictions as a self-guidance direction during sampling. To train this head, we further find that real images are not necessary. Instead, model-generated samples suffice and even outperform real images for training the head, especially in enhancing the high-frequency components that pixel diffusion tends to underfit. Across multiple pixel diffusion models on ImageNet, our \textbf{Synthetic Self-Guidance (SSG)} consistently improves generation while adapter training requires less than 1$\%$ of full-model training compute: it reduces FID by over 50$\%$ across the evaluated JiT variants without classifier-free guidance (CFG) and further improves strong baselines with CFG, e.g., JiT-H/16 from 1.86 to 1.67 and PixelREPA-H/16 from 1.81 to 1.59. Our code is available at https://github.com/zfu006/SSG.

Introduction

Synthetic Self-Guidance (SSG) improves pixel-space diffusion by using a frozen pretrained model’s intermediate coarse predictions to guide final high-frequency refinement. The adapter is trained on model-generated samples rather than real images, avoiding backbone retraining while addressing blurry textures and missing detail.

  • Motivation: Pixel-space diffusion avoids a VAE-based latent pipeline but must model global semantics and fine-grained details jointly in a high-dimensional pixel space.This modeling burden motivates methods that can improve a pretrained pixel diffusion model without training a replacement from scratch.
  • Method: SSG uses the discrepancy between intermediate coarse predictions and final refined predictions as guidance during sampling, without retraining the pretrained backbone.Intermediate layers capture low-frequency structure, while final layers progressively refine local high-frequency details.
  • Contribution: SSG demonstrates that a frozen pixel diffusion transformer can use its own learned representations to improve generation through a lightweight plug-in guidance mechanism.The approach exploits the model’s internal coarse-to-fine progression rather than requiring a newly designed model trained from scratch.
  • Adapter training: The adapter is trained on model-generated samples instead of real images, because it only needs to provide a coarse low-frequency reference for effective self-guidance.Generated-sample training helps alleviate blurry textures and missing high-frequency detail in final images, and is reported as more effective than training on real images.

Related Work

Related work contrasts latent- and pixel-space diffusion, develops sampling-time guidance, and increasingly uses model-generated samples to improve pretrained generators. SSG differs by using synthetic samples to train an intermediate coarse-prediction adapter grounded in the backbone’s coarse-to-fine structure.

  • Latent- and Pixel-Space Diffusion: Latent diffusion models use VAEs to reduce training cost, while DiT and SiT replace U-Nets with transformer-based architectures and flow-matching-related objectives (Rombach et al. 2022; Peebles and Xie 2023; Ma et al. 2024; Ho, Jain, and Abbeel 2020; Liu, Gong, and Liu 2022).These methods operate in compressed latent spaces or alter the diffusion architecture and prediction formulation.
  • Latent- and Pixel-Space Diffusion: Pixel-space diffusion denoises raw pixels, motivating architectures such as JiT, DeCo, DiP, and PixelDiT to reduce the computational and modeling difficulty of high-dimensional images (Ho, Jain, and Abbeel 2020; Li and He 2026; Dosovitskiy et al. 2020; Ma et al. 2026; Chen et al. 2026; Yu et al. 2026).JiT uses a plain ViT with large patches and directly predicts clean images, while later methods introduce hierarchical designs.
  • Sampling Guidance: Sampling guidance improves diffusion quality through classifier-free guidance and related methods such as Autoguidance, which replaces unconditional branches with a weak model’s output (Ho and Salimans 2022; Karras et al. 2024).CFG extrapolates from an unconditional prediction toward a conditional prediction, whereas Autoguidance uses a weak model as the reference.
  • Synthetic-Sample Guidance: Unlike SIMS and Neon, which use self-synthesized samples for auxiliary score training or weight-space negative extrapolation, SSG trains a lightweight intermediate-layer adapter while freezing the backbone (Alemohammad et al. 2024; Alemohammad, Wang, and Baraniuk 2025).SSG uses synthetic samples as training data for coarse intermediate predictions, then guides final outputs with those predictions during sampling.

Method

Synthetic Self-Guidance (SSG) improves a frozen pretrained pixel diffusion model by decoding an intermediate representation into a coarse prediction and using it to guide the final prediction during sampling. The adapter is trained only on model-generated samples, updates no backbone parameters, and requires less than 1% of full-model training compute.

  • Synthetic Self-Guidance: SSG decodes an intermediate frozen-backbone representation with a lightweight adapter and uses its coarse prediction as the weak prediction for guiding the final prediction.The intermediate and final predictions are derived from the same pretrained backbone, avoiding a separate weak model or joint backbone training.
  • Observation and Motivation: 99.5% low-frequency band power at Layer 4 contrasts with 87.7% high-frequency band power, showing that intermediate predictions capture coarse structure before refining local texture.Low-frequency power is already close to the final prediction at shallow layers, while high-frequency power increases with depth.
  • Synthetic Self-Guidance: Self-guidance extrapolates the final prediction away from the intermediate prediction when w > 1, while w = 1 recovers the original final output.The shared components are preserved while the discrepancy between intermediate and final predictions supplies the guidance direction.
  • Training the Adapter with Synthetic Samples: The adapter is trained entirely on samples generated by the pretrained model, because self-guidance requires only a coarse low-frequency counterpart rather than a full reconstruction of real images.SSG therefore requires no real images and updates only the adapter.
  • Training the Adapter with Synthetic Samples: SSG uses less than 1% of full-model training compute and can operate alone or alongside classifier-free guidance.When combined with CFG, self-guidance is applied independently to conditional and unconditional clean predictions before converting them into velocities.

Experiments

SSG improves class-conditional ImageNet generation across frozen pixel diffusion backbones, including JiT, PixelREPA-H, and hierarchical DeCo-XL, while requiring less than 1% of full-model training compute. Synthetic samples outperform real images for adapter training, and ablations show that the method remains effective with compact adapters, few synthetic samples, and varied guidance settings.

  • A Lightweight Adapter Suffices: Adapter training costs less than 1% of full-model training, using one transformer block for base and large JiT models and two blocks for huge models, while one block provides a favorable quality-cost tradeoff.A linear adapter performs worse than the JiT baseline and causes patch-boundary artifacts; adding a transformer block removes these artifacts, while two blocks slightly improve FID but exceed 1% compute on JiT-B/16.
  • Class-Conditional Image Generation: SSG reduces FID from 1.86 to 1.67 for JiT-H/16, 1.94 to 1.84 for JiT-H/32, and 1.81 to 1.59 for PixelREPA-H/16, while improving DeCo-XL/16 from 1.69 to 1.63 with CFG.These improvements hold across the evaluated backbones and transfer to the hierarchical DeCo architecture without per-layer tuning; IS decreases slightly because scales are selected for lowest FID.
  • Synthetic Samples Outperform Real Images for Adapter Training: Synthetic adapter training lowers JiT-H/16 FID from 2.51 to 2.26 without CFG and from 1.78 to 1.67 with CFG, outperforming real-image training across nearly all settings.Synthetic-trained adapters also produce samples with radial power closer to real images than real-trained adapters.
  • Fewer Synthetic Samples Suffice: Reducing synthetic data from 1M to 10K samples changes FID by less than 0.05, from 3.29 to 3.31 for JiT-B/16 and from 3.67 to 3.69 for JiT-B/32.The comparison keeps total training iterations fixed, while the main experiments use 1M samples to match the scale of ImageNet training data.
  • Early-to-Middle Layers Provide the Best Guidance: Early-to-middle layers provide the strongest guidance, with Layers 4–6 performing comparably while deeper Layers 7–8 approach the final prediction and sharply degrade toward baseline quality.Compared with jointly trained IG, SSG reaches 3.29 FID versus 3.41 for IG and 3.50 for a frozen adapter trained on real images.
  • Effect of Guidance Scales: SSG improves the baseline across broad guidance-scale combinations, reducing JiT-B/16 FID from 3.66 to 3.33 at the baseline CFG scale of 3.0 with SSG scale w = 1.2.Changing the CFG used to generate synthetic data leaves downstream FID nearly unchanged, within 0.06 for JiT-B/16 and 0.05 for JiT-B/32.

Conclusion

Synthetic Self-Guidance (SSG) improves pretrained pixel diffusion models while keeping their backbones frozen. It uses intermediate-to-final prediction discrepancies for sampling guidance and trains the adapter solely on model-generated samples, which outperform real images in nearly all evaluated settings.

  • Conclusion: SSG is a plug-in method that improves pretrained pixel diffusion models while keeping their backbones frozen.The method is designed to adapt existing models without updating the backbone.
  • Conclusion: SSG decodes an intermediate representation into a coarse prediction and uses its discrepancy from the final prediction as self-guidance during sampling.This discrepancy provides the guidance direction used during generation.
  • Conclusion: Model-generated samples train the adapter solely and outperform real images for this purpose in nearly all evaluated settings.The conclusion reports this synthetic-data advantage across the evaluated settings.

Supplementary Material · A Implementation Details

The supplementary implementation details specify adapter initialization and training, optimizer settings, and sampling configurations for JiT, PixelREPA, and DeCo. Adapter training updates only the lightweight adapter, while sampling uses Heun integration with model-specific noise and guidance settings.

  • A Implementation Details: Table S2 summarizes the backbone, adapter, training, and sampling configurations used in the experiments.The table provides the consolidated implementation settings for the evaluated systems.
  • A Implementation Details: Adapters are initialized from late pretrained backbone blocks and the pretrained final prediction head, while DeCo copies its pixel decoder as the adapter output layer.For JiT and PixelREPA, adapter blocks use the late backbone blocks listed in Table S2; DeCo uses its last two encoder blocks.
  • A Implementation Details: Only the adapter parameters are updated during adapter training.The frozen backbone is not optimized during this stage.
  • A Implementation Details: Training uses Adam with β1 = 0.9 and β2 = 0.95, without weight decay or learning-rate warmup.JiT and PixelREPA use learning rate 2e-4 with global batch size 1024 after scaling, whereas DeCo uses 5e-5 with batch size 256.
  • A Implementation Details: JiT and PixelREPA adapters predict the clean image, whereas DeCo directly predicts velocity without representation supervision or auxiliary frequency loss.These objectives are used during adapter training across the evaluated model families.
  • A Implementation Details: Sampling uses the Heun sampler for 50 steps across all models.This common sampler configuration is used for the reported experiments.
  • A Implementation Details: JiT uses noise scales 1.0 at 256 resolution and 2.0 at 512, while PixelREPA and DeCo use 1.0.The noise scale is therefore resolution-dependent for JiT but fixed for the other two model families.
  • A Implementation Details: Without CFG, SSG is applied throughout sampling; with CFG, both guidance methods operate within t ∈[0.1, 1.0].The model- and resolution-specific CFG and SSG scales are provided in Table S1.

B Guidance-Scale Results

SSG improves FID across broad CFG and SSG guidance-scale ranges, indicating robustness to scale choice. At the original baselines’ CFG scales, w = 1.1 lowers FID on both JiT-H/16 and JiT-H/32.

  • B Guidance-Scale Results: FID drops from 1.86 to 1.73 on JiT-H/16 and from 1.94 to 1.89 on JiT-H/32 at the original baselines’ CFG scales with w = 1.1.This fixed-CFG comparison shows the gains do not merely come from searching additional guidance-scale combinations.
  • B Guidance-Scale Results: SSG improves FID across a broad range of CFG and SSG scales, rather than depending on one specific scale combination.Figure S1 reports the full guidance-scale sweeps for JiT-H/16 and JiT-H/32.

C Frequency-Domain Analysis Details

The frequency-domain analysis examines layer-wise predictions and generated images using orthonormal Fourier transforms with channel-aggregated power. It compares JiT-B/16 baseline and self-guided samples trained on real or synthetic images under controlled sampling conditions.

  • Layer-Wise Band Power: Layer-wise analysis compares clean predictions from adapters at Layers 4, 6, and 8 of JiT-B/16 with the final prediction at noise level t = 0.5.It uses 512 ImageNet validation images and groups Fourier coefficients by frequency after transforming each RGB channel.
  • Generated-Image Radial Power: Generated-image analysis compares 2048 class-balanced ImageNet validation images with JiT-B/16 baseline samples and self-guided samples using adapters trained on real or synthetic images.Radial power is computed from orthonormal Fourier transforms with channel aggregation and averaged over the resulting power maps.
  • Generated-Image Radial Power: All generated comparisons use matched class labels and initial noise, with real and generated images clamped to [0, 1].This standardizes the sampling inputs and image range before radial-power computation.

D Additional Qualitative Results

Additional class-conditional samples from JiT-H/16 with SSG further demonstrate that the method generates high-quality images.

  • D Additional Qualitative Results: SSG produces high-quality class-conditional samples with JiT-H/16, as shown in Fig. S2, Fig. S3, and Fig. S4.

E Limitations and Future Work

SSG is demonstrated primarily for class-conditional ImageNet generation with pixel-space diffusion models, leaving its broader generalization uncertain. Future work should test text-to-image generation, broader datasets, and other diffusion architectures.

  • SSG is evaluated mainly on class-conditional ImageNet generation with pixel-space diffusion models.
  • Whether intermediate-to-final refinement generalizes to larger text-conditioned models or domains with different image distributions remains unclear.
  • Future work could extend SSG to text-to-image generation, broader datasets, and other diffusion architectures.
Loading 2607.29122v1…