PaperlayerView exact revision on arXiv

Source-linked AI summary

Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusion

Weichen Fan, Haiwen Diao, Penghao Wu, Ziwei Liu

arXiv:2606.15236v2cs.CV

TL;DR

Pixel-space diffusion denoisers must internally discover a moving signal-to-noise bandwidth boundary, potentially allocating computation to noise-dominated frequencies. Spectral Forcing makes that boundary explicit with a time-conditional DCT low-pass, improving ImageNet-256 FID and Inception Score at coarse tokenization while remaining competitive at finer tokenization.

  • Problem

    Pixel-space denoisers must discover the timestep-dependent signal-to-noise bandwidth boundary internally, creating a capacity-allocation challenge.

  • Method

    Spectral Forcing applies a parameter-free, time-conditional 2D-DCT low-pass before patch embedding, with a monotonically expanding cutoff that becomes identity at the data endpoint.

  • Results

    FID drops from 24.19 to 20.68 and Inception Score rises from 83.28 to 93.96 on ImageNet-256 with JiT-700M/32, while remaining competitive at finer tokenization.

  • Takeaways & Limitations

    Spectral Forcing provides a capacity-efficient input-side frequency prior for pixel-space diffusion within the identified coarse-tokenization, noise-dominated regime.

  • Takeaways & Limitations

    Its largest gains require coarse patch tokenization and high-frequency content dominated by noise rather than essential signal.

Abstract

from arXiv · show

Pixel-space diffusion models are trained on full-bandwidth noisy images, yet the useful signal available to the denoiser is strongly frequency dependent. Under rectified-flow diffusion and natural-image power-law spectra, the per-band data-to-noise contour $k^{*}(t) = (1-t)^{-2/α}$ separates a signal-bearing low-frequency region from a noise-dominated high-frequency region at each time $t$. We show that this implicit coarse-to-fine structure is not merely descriptive: it induces a capacity-allocation problem. A standard pixel-space denoiser must discover the moving bandwidth boundary internally and can spend computation on frequency-time regions where the optimal prediction collapses to deterministic baselines rather than data-distribution modeling. To make this boundary explicit, we introduce Spectral Forcing, a parameter-free, time-conditional 2D-DCT low-pass operator applied to the noisy input before the patch embedder. Its cutoff expands monotonically with the diffusion time and becomes the identity at the data endpoint. Through controlled synthetic experiments, we identify the regime in which the operator is beneficial: coarse patch tokenization and data whose high-frequency content is predominantly noise rather than essential signal. On ImageNet-256 with JiT-700M/32, Spectral Forcing consistently improves both FID and Inception Score across different training epochs, demonstrating robust gains throughout training; at finer tokenization, the spectral forcing is still competitive. We further insert the unchanged operator into SenseNova-U1, a unified text-to-image model, where it improves DPG-Bench and GenEval, showing that the input-side spectral prior transfers beyond class-conditional generation. These results suggest a route to capacity-efficient pixel-space diffusion by showing the signal and hiding the noise.

1 Introduction

Spectral Forcing makes diffusion’s moving signal–noise bandwidth boundary explicit with a time-conditional DCT low-pass before patch embedding. It improves pixel-space generation most when coarse tokenization already limits bandwidth, while delivering robust ImageNet gains and transferring to unified text-to-image models.

  • Motivation: Diffusion is implicitly coarse-to-fine because timestep-dependent noise buries high-frequency signal, forcing denoisers to reconstruct low frequencies before high frequencies.The paper argues this hierarchy has rarely been exploited as an explicit architectural prior.
  • Motivation: The contour k*(t) = (1 −t)^−2/α separates signal-bearing frequencies from noise-dominated regions where prediction collapses to deterministic baselines.A full-bandwidth denoiser must discover this moving boundary internally, potentially wasting capacity on frequency-time regions that require little data-distribution modeling.
  • Method: Spectral Forcing applies a parameter-free, time-conditional 2D-DCT low-pass before the patch embedder, with a cutoff that grows monotonically with time.The fixed-by-design schedule is imposed at the input rather than estimated from the data spectrum.
  • Regime: Spectral Forcing delivers its largest gains when patch tokenization is coarse and high-frequency content is predominantly noise rather than essential signal.The paper identifies this regime in controlled toy experiments and on ImageNet.
  • Results: FID from 24.19 to 20.68 (+14%) and Inception Score from 83.28 to 93.96 (+13%) with JiT-700M/32 on ImageNet-256 in a 60-epoch comparison.The improvement was consistent for both metrics across evaluated training-epoch checkpoints.
  • Implications: The results suggest a capacity-efficient route to pixel-space diffusion by showing denoisers the signal while hiding the noise.The unchanged input-side operator also transfers beyond class-conditional generation to a unified text-to-image model, improving DPG-Bench and GenEval.

2 Related Work

Prior work spans latent and pixel-space diffusion, while spectral approaches modify the forward process, analyze implicit coarse-to-fine structure, or generate in frequency representations. This work is positioned within pixel-space generation and frequency-domain methods.

  • Diffusion and pixel-space generation: Diffusion and flow matching dominate high-quality image generation, with transformer backbones increasingly replacing U-Nets and latent diffusion using separately trained autoencoders.Recent latent-diffusion work also explores representation-rich tokenizers for faster convergence, while pixel-space diffusion provides an alternative.
  • Spectral and frequency-domain methods: Spectral methods include progressive blurring, wavelet shrinkage, analyses of implicit coarse-to-fine behavior, and direct generation in frequency representations.These analyses connect diffusion’s coarse-to-fine character with neural-network spectral bias and natural-image power-law statistics.

3 Methodology

The methodology identifies a moving signal/noise boundary in rectified-flow diffusion and shows that denoisers waste capacity outside a low-frequency signal-recovery wedge. Spectral Forcing makes this boundary explicit with a time-conditional, parameter-free DCT low-pass whose benefits depend on patch granularity and the data spectrum.

  • Rectified-flow formulation: Rectified flow interpolates noise and data as z_t = t x + (1 − t) ε, with a denoiser trained by squared-error flow matching.Sampling uses logit-normal training times, EMA inference weights, Heun integration, and classifier-free guidance.
  • Per-band data-to-noise ratio: The natural-image spectrum yields a bandwidth front k*(t) = (1 − t)^−2/α, separating signal-bearing low frequencies from noise-dominated high frequencies.ImageNet-256 has an estimated spectral exponent α ≈ 2.82, and the front expands as t approaches 1.
  • Empirical motivation: The denoiser performs learned data-distribution work only inside a low-k signal-recovery wedge, while other regions converge toward closed-form denoising or a zero-predictor baseline.The same three-region arrangement appears in a JiT-700M/32 ImageNet checkpoint at 60 ep, including a high-t predict-zero region where log10(MSEnet/MSEzero) ≥ 0.
  • Regime of benefit: An input-side low-pass helps most with coarse patchification and spectra whose high-frequency content is predominantly noise, but it removes essential edge signal from rectangle data.On power-law data, analytical L1 improves 26.2→16.5 and linear improves 26.2→23.1; on rectangles, baseline 26.8 worsens to linear 47.1 and analytical 40.9.
  • The operator and schedule: Spectral Forcing applies a parameter-free DCT mask before network processing, with a monotonic cutoff that is aggressive at t = 0 and becomes the identity at t = 1.The analytical schedule tracks k*(t) exactly, while linear-SF is the reported default because it performs better on ImageNet at 64 tokens.

4 Experiments

Experiments show that Spectral Forcing’s benefits are strongest with coarse tokenization, persist across training budgets at large scale, and depend on a time-dependent DCT schedule. Alternative operators underperform, while the unchanged operator is also evaluated in a unified text–image model.

  • Cross-scale picture: Spectral Forcing reduces FID across every tested model-and-epoch pair with coarse tokenization, while its effect at 256 tokens is within evaluator noise.The coarse-tokenization advantage is largest at 64 tokens and remains neutral at 256 tokens across all tested epoch budgets.
  • Effect of training budget: +14.5% is the largest reported gain, achieved by JiT-700M/32 with 60 training epochs under coarse tokenization.At JiT-130M/32, the margin falls from +11.6% at 15 epochs to +0.4% at 100 epochs, then retains +1.5% at 200 epochs.
  • Effect of training budget: At JiT-700M/32, the gain compresses from +14.5% at 60 epochs to +8.0% at 120 epochs, whose SF FID of 15.15 matches the previous-best 700M+SF reference at ∼145 epochs.This indicates that the large-scale benefit persists substantially farther into training than the small-scale margin.
  • Comparison to alternative operators: Constant low-pass, Gaussian blur, Focal Frequency Loss, Blurring diffusion, and DCTDiff all lose to the unforced baseline, supporting time-dependent input-side DCT masking.The alternatives test time dependence, spatial versus frequency-domain masking, loss-side reweighting, altered forward diffusion, and full DCT-domain modeling.
  • Native vision-language models: The unchanged Linear-SF operator is inserted into SenseNova-U1 to test transfer to a unified text–image model operating directly on raw image patches.This setting matches the coarse-tokenization regime where Spectral Forcing delivers its largest gains and where token counts must remain small for joint sequence modeling.

5 Ablation Study

The ablations show that Spectral Forcing is beneficial in a bounded regime determined mainly by token count, image resolution, and data structure. Its gains persist beyond early data efficiency in favorable settings, while schedule choice, compute overhead, and fine-tokenization limits explain where it helps or fails.

  • Regime boundaries: At 64 tokens, the operator helps across backbone sizes at 256^2, but higher token counts reverse the outcome unless resolution increases.Toy experiments hold token count at 64 with p=h/8; the favorable regime is bounded by tokenization and resolution.
  • Data distribution: On power-law data the baseline catches up at convergence, structured data favors the analytical schedule, and rectangles defeat both schedules because edges remain high-frequency signal.The results connect Spectral Forcing’s benefit to high-frequency content being predominantly noise rather than essential signal.
  • Image resolution: On toys, analytical-SF moves from worst at h=64 to best at h≥128, reaching −15% at h=256 and −3.3% at h=512.On real images, JiT-130M/32 at 512^2 recovers +3.4% FID after being neutral at 256^2.
  • Patch size: At p=32, Spectral Forcing reduces FID by +3.9% at 130M and +14.5% at 700M, while p=16 and p=64 remain within evaluator noise.The combined resolution results identify token count, rather than patch size or image resolution alone, as the operative axis.
  • Schedule choice and limitations: At 256 tokens, converged SF schedules sit within 0.53 FID points of baseline, but analytical-SF loses −6.6% class diversity, with IS 78.04 versus 83.59.At 64 tokens, linear beats analytical by 1.3 FID, reflecting patchify bandlimiting and training-dynamics effects.
  • Training budget: At JiT-700M/32, the margin compresses from +14.5% to +8.0% over 60–120 epochs, while 120-epoch SF FID 15.15 matches the published ∼145-epoch reference FID 15.24.The toy 2000-epoch block widens the analytical-SF margin from −15% to −17% at h=256, showing gains are not purely data-efficiency artifacts.
  • Efficiency: SF adds ≈0.5% per-step compute and reaches JiT-700M/32 baseline FID targets in 60/90/120 epochs versus 90/120/145 epochs, reducing wall-clock time by 17–33%.Inference cost is unchanged apart from the ≈0.5% DCT overhead.

6 Conclusion

Spectral Forcing makes diffusion’s implicitly learned bandwidth boundary explicit through a parameter-free, time-conditional DCT low-pass prior, improving ImageNet-256 generation quality with negligible compute overhead.

  • 6 Conclusion: Spectral Forcing applies a parameter-free, time-conditional 2D-DCT low-pass before the patch embedder, using a cutoff derived from the rectified-flow data-to-noise contour.It converts the bandwidth boundary learned implicitly during diffusion training into an explicit input-side prior.
  • 6 Conclusion: At JiT-700M/32 on ImageNet-256, Spectral Forcing improves both FID and Inception Score.The operator composes with any pixel-space recipe at negligible compute overhead.

A Implementation Details

The implementation adds Spectral Forcing as a deterministic, input-side adapter to an otherwise unmodified JiT pipeline. Experiments use fixed operator settings, matched DCT and patch sizes, specified time sampling, and standardized ImageNet evaluation.

  • Backbone and patchify: Experiments follow the JiT recipe while keeping the JiT backbone unmodified across JiT-130M/32, JiT-130M/16, and JiT-700M/32.The DCT window is matched to the patch-embedder patch size.
  • Time distribution: Training uses logit-normal diffusion times with μ = −0.8 and σ = 0.8 on ImageNet-256 throughout.Times are sampled by drawing s ∼ N(μ, σ^2) and setting t = sigmoid(s).
  • Spectral Forcing operator: Spectral Forcing applies a single soft global 2D-DCT radial low-pass to rectified-flow input z_t before the patch embedder.The operator is parameterized by cutoff endpoints, schedule shape, and soft-mask sharpness.
  • Spectral Forcing operator: The paper fixes c_min = 0.05, c_max = 1.0, κ = 30, and uses the linear schedule f(t) = t except in toy resolution-scaling experiments.The analytical schedule f(t) ∝ (1 − t)^−2/α is used only for those toy experiments.
  • Evaluation: ImageNet FID-50k uses 50-step Heun sampling with classifier-free guidance scale 2.9 and CFG interval [0.1, 1.0], while Inception Score uses the same 50k images.Toy experiments report radial-spectrum L1 distance to the empirical data spectrum.

B Additional Experiments · B.1 Hyperparameter sensitivity: the cmin sweep.

The cmin sweep shows that Spectral Forcing’s convergence loss varies continuously with operator restrictiveness. Less aggressive masking brings SF closer to the baseline at convergence.

  • B.1 Hyperparameter sensitivity: the cmin sweep.: cmin = 0.40 → L1 = 10.69, the lowest loss in the sweep at the canonical toy setting.The setting uses h = 64, p = 8, α = 2, linear-SF, and 1000 epochs.
  • B.1 Hyperparameter sensitivity: the cmin sweep.: cmin = 0.00 → L1 = 17.43, while cmin = 0.30 → 10.95, showing lower loss with less restrictive masking.Intermediate values are cmin = 0.10 → 14.42 and cmin = 0.20 → 14.66.
  • B.1 Hyperparameter sensitivity: the cmin sweep.: The monotonic cmin sweep indicates that SF’s loss at convergence is continuous rather than a discrete failure mode.Larger cmin values make the operator less aggressive and bring SF closer to the baseline at convergence.

B.2 Schedule shapes at h = 128. · B.3 Resolution-scaling: per-seed values at h = 256.

At h = 128, aggressively early-cutting schedules perform best at convergence, while a 2000-epoch check confirms analytical-SF’s advantage over baseline and linear-SF. At h = 256, both Spectral Forcing schedules beat baseline with ±2σ separation despite per-seed variance.

  • B.2 Schedule shapes at h = 128.: At h = 128 with p = 16 and n = 4 seeds, schedules that aggressively cut early bands, including analytical and t2, win at convergence.This ordering is described as the clean opposite of h = 64.
  • B.2 Schedule shapes at h = 128.: At h = 128, schedules that are over-permissive at small t lose, while cosine is roughly tied.
  • B.2 Schedule shapes at h = 128.: The standard deviation across seeds is small at 1–2 L1 units, and the analytical-wins gap is much larger than seed variance.
  • B.2 Schedule shapes at h = 128.: At 2000 epochs and h = 128, baseline records 32.86, linear-SF records 35.75, and analytical-SF records 28.37.This sanity check uses a single seed and confirms the schedule ordering.
  • B.3 Resolution-scaling: per-seed values at h = 256.: At h = 256 with p = 32, both Spectral Forcing schedules beat baseline at ±2σ separation.Table 11 reports the per-seed values for this configuration.
  • B.3 Resolution-scaling: per-seed values at h = 256.: Additional seeds were not run at h = 256 because the per-seed gap was much larger than the per-seed variance.

B.4 SenseNova-U1: GenEval breakdown. · B.5 Closed-form denoising limit. · B.6 Algorithmic listings.

SenseNova-U1’s GenEval improves with Spectral Forcing, especially for single-object and color prompts, while the closed-form limit shows that high-frequency denoising can reduce to deterministic rescaling. The appendices also specify cutoff construction and masked training and sampling procedures.

  • B.4 SenseNova-U1: GenEval breakdown.: GenEval rises from 3.87% to 4.56% (+17.9% relative) with Spectral Forcing at the matched SenseNova-U1 checkpoint.Per-correct-image and per-correct-prompt percentages also increase.
  • B.4 SenseNova-U1: GenEval breakdown.: Single-object and colors show the largest gains, increasing by +2.81 pp (+19.1%) and +1.33 pp (+15.6%), respectively.The improvements are concentrated in these two categories.
  • B.4 SenseNova-U1: GenEval breakdown.: Two-object, counting, position, and color-attr score 0% for both methods at the early checkpoint and are therefore omitted.These compositional categories require a later checkpoint where compositionally correct outputs begin to appear.
  • B.4 SenseNova-U1: GenEval breakdown.: The GenEval result, together with the DPG-Bench breakdown, indicates that the input-side spectral prior transfers to native-VLM text-to-image generation in its predicted favourable regime.This conclusion concerns SenseNova-U1 generation.
  • B.5 Closed-form denoising limit.: In that region, the denoising target is a deterministic function of the input: pure rescaling uses nothing about the data distribution.The result follows from the rectified-flow interpolant zt = t x + (1 − t) ε.
  • B.5 Closed-form denoising limit.: In high-k bands where P(k) ≪ 1 − t, the signal is negligible, so zt,k ≈ (1 − t) εk and εk ≈ zt,k/(1 − t).This identifies the noise-dominated region at t bounded away from 1.
  • B.6 Algorithmic listings.: Algorithm 1 defines cutoff schedules, Algorithm 2 constructs the soft 2D-DCT radial low-pass M(t), and Algorithms 3–4 modify JiT training and Euler sampling steps.Class conditioning and CFG are omitted for brevity.

B.7 Qualitative samples.

Qualitative samples from JiT-700M/32 at 120 epochs compare matched ImageNet generations with and without Spectral Forcing. The method improves FID and produces more specific textures, especially on textured surfaces.

  • Qualitative samples: At 120 epochs, matched ImageNet samples from JiT-700M/32 compare the no-mask baseline (FID 16.46) with Linear-SF (FID 15.15, +8.0%).Each pair fixes the class label and sample index, isolating whether the time-conditional 2D-DCT low-pass was active during training and sampling.
  • Qualitative samples: The qualitative comparison spans birds, mammals, marine subjects, and prepared food across nine ImageNet classes.The listed classes include lorikeet, indigo bunting, golden retriever, lion, stingray, coral reef, and pizza.
  • Qualitative samples: Spectral Forcing is most visibly beneficial on textured surfaces, where the no-mask baseline produces smoother but less specific texture.Examples include lion mane, coral, pizza topping, and cliff dwelling stonework.

C Limitations

The main limitations are modest per-step overhead, fixed hyperparameters, and evaluation concentrated on one benchmark and architecture family. Spectral Forcing remains parameter-free and can be removed when its compute cost matters.

  • Per-step compute overhead: 0.5% of per-step compute relative to the unmasked baseline at JiT-130M/32, 2562 is added by one forward and one inverse 2D-DCT per denoising step.The DCT uses no learned parameters or additional memory, is parallelizable on the patch grid, and runs in the same kernel as patchify.
  • Per-step compute overhead: For budgets where 0.5% matters, Spectral Forcing can be made a no-op to remove its overhead.This preserves the option to eliminate the operator without changing its learned-parameter or memory profile.
  • Fixed hyperparameters: The cutoff bounds (cmin, cmax)=(0.05, 1.0) and mask sharpness κ=30 are reused unchanged across every ImageNet configuration.No per-model, per-budget, or per-resolution tuning is used; the cmin sweep indicates the operator is not narrowly tuned at this point.
  • Single benchmark and architecture family: Evaluation is reported on ImageNet-256 and JiT, leaving other backbones and diffusion forwards as untested configuration changes.The stated examples of other backbones are DiT and U-Net, while the reported architecture is a pixel-space rectified-flow Transformer.
Loading 2606.15236v2…