Source-linked AI summary
Pixel-Space Diffusion via Observation Operators
Shaojie Guo, Lichen Ma, Haoyang Tong, Yu He, Zipeng Guo, Xiaoan Liu, Feng Yan, Yu Guo, Fei Wang, Junshi Huang, Yan Wang
TL;DR
Pixel-space diffusion models supervise full images even when high-noise inputs cannot reliably reveal fine details, creating a scale–time mismatch. The paper introduces time-indexed Gaussian–Lanczos supervision and coarse-to-fine decoder refinement, achieving FID 1.52 on ImageNet-256 with faster convergence and improved generation quality.
Problem
Pixel-space diffusion models use fixed full-image supervision despite coarse-to-fine structure recoverability, creating a scale–time mismatch that hinders optimization.
Method
Observation Operator Diffusion aligns denoising supervision and decoder feature refinement with structure recoverability using time-indexed Gaussian–Lanczos observations and GL-CoDA.
Results
FID 1.52 on ImageNet-256, with substantially faster convergence and consistently improved generation quality.
Takeaways & Limitations
The results support coarse-to-fine alignment as a way to improve optimization efficiency and generation quality in pixel-space diffusion.
Takeaways & Limitations
The observation-operator path assumes operators that preserve spatial resolution while controlling the spatial scale of observed content.
Abstract
from arXiv · showhide
Pixel-space diffusion models directly model image distributions but remain difficult to optimize. Recent methods alleviate this challenge through target reparameterization, while still relying on a fixed clean-image target throughout denoising. Through empirical analysis, we identify a scale-time mismatch: image structures become predictable from coarse to fine as noise decreases, whereas existing models are forced to predict the full image even under high noise, resulting in low-SNR gradients that hinder optimization. To resolve this mismatch, we propose Observation Operator Diffusion, a unified framework that aligns both the supervision trajectory and feature refinement with the intrinsic recovery order of image structures. Specifically, we replace fixed full-image supervision along the standard flow path with a time-indexed observation trajectory that evolves from coarse structures to the full image during denoising. This trajectory is instantiated with a family of Gaussian-Lanczos operators at varying observation scales, yielding a path-consistent training objective. We further introduce GL-CoDA, a decoder that injects scale-specific Gaussian-Lanczos observations across decoding stages for coarse-to-fine feature refinement. Extensive experiments show that the proposed approach converges substantially faster while consistently improving generation quality, achieving an FID of 1.52 on ImageNet-256.
Introduction
Pixel-space diffusion avoids compression-related information loss but suffers from a scale–time mismatch: coarse structures become recoverable before fine details, while models retain fixed full-image supervision. Observation Operator Diffusion addresses this mismatch with time-indexed Gaussian–Lanczos observations, a path-consistent objective, and GL-CoDA, achieving faster convergence and an FID of 1.52 on ImageNet-256.
- Pixel-space diffusion models images directly, avoiding autoencoder compression and fixed-decoder reconstruction errors but creating substantially greater optimization difficulty.
- Scale–time mismatch arises because coarse structures become recoverable under high noise, whereas fine details become predictable only as noise decreases.
- Observation Operator Diffusion replaces fixed full-image supervision with a time-indexed trajectory of Gaussian–Lanczos observations that follows coarse-to-fine structure recovery.
- The path-consistent objective aligns supervised scale with recoverable structure at each denoising step, preventing unresolved fine details from interfering at high noise and producing higher-SNR gradients.
- GL-CoDA extends coarse-to-fine processing across network depth by injecting scale-specific Gaussian–Lanczos responses across decoding stages for feature refinement.
- FID 1.52 on ImageNet-256 demonstrates the method’s reported generation quality, alongside substantially faster convergence and consistent quality improvements.
Related Work
Related work spans pixel-space diffusion objectives, scale-aware generation methods, and classical scale-space image operators. This work connects these areas by making spatial scale a diffusion-time-indexed prediction target that evolves during denoising.
- Pixel-Space Diffusion and Prediction Objectives: Pixel-space diffusion avoids latent-autoencoder reconstruction bottlenecks but remains challenging to optimize, motivating alternative architectures and prediction objectives.Recent architectures explore patch–pixel modeling, hyper-connected feature reuse, and U-shaped transformer designs.
- Scale-Aware Pixel Diffusion: Scale-aware pixel diffusion methods model multiscale generation through frequency-aware losses, separate frequency transport paths, or DCT-domain input filtering.DeCo separates low-frequency semantics from high-frequency details; FREPix uses distinct low- and high-frequency transport paths; Spectral Forcing filters noisy inputs in the DCT domain.
- Scale-Space Representations and Image Operators: Scale-space theory represents images as continuously indexed observations across spatial scales, using operators such as Gaussian filters, Laplacian pyramids, and Lanczos kernels.These operators have long supported image representation, analysis, and resampling across scales.
- Scale-Space Representations and Image Operators: The proposed method connects scale-space representations to diffusion by constructing Gaussian–Lanczos observations indexed by diffusion time as dynamic prediction targets throughout denoising.This extends the initial exploration of scale as an image-representation attribute in diffusion models.
Method
Observation Operator Diffusion replaces fixed full-image supervision with a time-indexed Gaussian–Lanczos observation trajectory that follows coarse-to-fine image recovery. It extends this principle to feature refinement through a two-stage Patch–Pixel backbone with a GL-CoDA decoder.
- Patch–Pixel backbone: The architecture combines a Patch-DiT encoder with a GL-CoDA decoder whose GL structure branch injects scale-specific responses across decoding stages.This extends coarse-to-fine refinement from supervision to learned feature processing.
- Observation trajectory: The method replaces the fixed full-image target with observations that progress from coarse structures to the full clean image during denoising.The trajectory satisfies y0 = Acoarse(x0) and y1 = x0, with time-dependent operators required to vary absolutely continuously.
- Path-consistent objective: The observation path modifies the standard flow-matching velocity target with the temporal-anchor term t ˙yt.The path is zt = tAt(x0) + (1−t)ϵ, and its velocity is ˙zt = yt − ϵ + t ˙yt.
- Path-consistent objective: Standard flow matching is recovered when the operator is the identity, making the observation fixed at yt = x0 and ˙yt = 0.This establishes the proposed framework as a generalization of the standard flow-matching path and velocity target.
- Gaussian–Lanczos operators: Gaussian–Lanczos operators combine sharper scale selectivity from Lanczos filtering with Gaussian suppression of ringing artifacts.At matched −3 dB cutoff, the combined kernel has a narrower transition band than Gaussian smoothing alone.
- Gaussian–Lanczos operators: The time-dependent scale schedule uses broader kernels early and progressively narrower kernels later to recover edges and textures toward the clean image.Both f(t) and σ(t) decrease toward identity-filtering limits as t increases.
Experiments
Experiments show that the observation-path method improves pixel-space generation quality at 256 × 256 and 512 × 512 while preserving diversity and fine detail. Ablations attribute these gains to scale-adaptive supervision, complementary Gaussian–Lanczos filtering, and coarse-to-fine depth assignment.
- Evaluation setup: The evaluation uses class-conditional ImageNet-1K generation at 256 × 256 and 512 × 512, reporting FID, sFID, IS, precision, and recall from 50K samples.Unless otherwise specified, models and ablations are trained for 80 epochs.
- Quantitative results: 1.52 gFID is achieved on ImageNet 256 × 256 after 260 epochs, improving on the 80-epoch result of 1.95 while using 60 fewer epochs than the previous best.sFID and IS further validate the improvement without sacrificing sample diversity.
- Quantitative results: 1.58 gFID is achieved at 512 × 512, improving the previous best pixel-space result of 1.81 by 12.7%, while recall rises from 0.67 to 0.69.The result demonstrates scalability at four times the pixel count.
- Qualitative results: Qualitative samples show category-faithful shapes, coherent layouts, and preserved fine structures at 256 × 256, alongside high-frequency detail and global consistency at 512 × 512.Examples include mushroom gills, water droplets, animal fur, and weathered wood.
- Observation-path ablation: The Gaussian–Lanczos observation path consistently reduces FID across three architectures, with Patch–Pixel plus GL-CoDA performing best.The results indicate complementary benefits from time-dependent target refinement and depth-wise structural refinement.
- Kernel design: 1.95 FID is obtained by combining Gaussian and Lanczos filtering, outperforming either component alone through sharper scale selectivity with reduced ringing artifacts.Lanczos sharpens scale selectivity, while Gaussian smoothing attenuates oscillatory sidelobes.
- Depth assignment: Coarse responses assigned to shallow blocks and progressively finer responses assigned to deeper blocks achieve the best FID, whereas single-scale or reversed ordering performs worse.This shows that structural injection benefits from matching coarse-to-fine ordering with decoder depth.
- Scale and schedule sensitivity: The final configuration f0 = 2, σ0 = 1, and a linear schedule achieves the best FID of 1.95 and is robust to f0 and schedule variations within the tested range.Performance is more sensitive to σ0.
Conclusion
The paper identifies a scale–time mismatch in pixel-space diffusion: image structures become predictable coarse-to-fine during denoising, but existing models supervise all spatial scales at every step. It addresses this mismatch by replacing fixed full-image supervision with a time-indexed observation trajectory.
- Conclusion: Pixel-space diffusion suffers from a scale–time mismatch because image structures follow a coarse-to-fine predictability order during denoising.Existing models supervise all spatial scales at every step, producing low-SNR gradients that slow convergence.
- Conclusion: The proposed observation-operator diffusion framework replaces fixed full-image supervision with a time-indexed observation trajectory.