PaperlayerView exact revision on arXiv

Source-linked AI summary

DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation

Zehong Ma, Longhui Wei, Shuai Wang, Shiliang Zhang, Qi Tian

arXiv:2511.19365v2cs.CVcs.AI

TL;DR

Pixel diffusion avoids VAE limitations but struggles to model high-frequency signals and low-frequency semantics efficiently in raw pixel space. DeCo separates these roles between a DiT and a lightweight pixel decoder, adds frequency-aware flow matching, and achieves leading pixel-diffusion results while narrowing the gap with latent diffusion.

  • Problem

    Pixel diffusion directly models raw pixels without a VAE, but jointly learning high-frequency signals and low-frequency semantics in the large pixel space is difficult and can degrade quality.

  • Method

    DeCo assigns low-frequency semantic modeling to a DiT, high-frequency reconstruction to a lightweight conditioned pixel decoder, and frequency weighting to a JPEG-inspired flow-matching loss.

  • Results

    DeCo achieves FID 1.62 at 256×256 and 2.22 at 512×512 on ImageNet, while its text-to-image model reaches 0.86 on GenEval.

  • Takeaways & Limitations

    DeCo achieves leading performance among pixel diffusion models on class-to-image and text-to-image benchmarks, closing the gap with two-stage latent diffusion methods.

Abstract

from arXiv · show

Pixel diffusion aims to generate images directly in pixel space in an end-to-end fashion. This approach avoids the limitations of VAE in the two-stage latent diffusion, offering higher model capacity. Existing pixel diffusion models suffer from slow training and inference, as they usually model both high-frequency signals and low-frequency semantics within a single diffusion transformer (DiT). To pursue a more efficient pixel diffusion paradigm, we propose the frequency-DeCoupled pixel diffusion framework. With the intuition to decouple the generation of high and low frequency components, we leverage a lightweight pixel decoder to generate high-frequency details conditioned on semantic guidance from the DiT. This thus frees the DiT to specialize in modeling low-frequency semantics. In addition, we introduce a frequency-aware flow-matching loss that emphasizes visually salient frequencies while suppressing insignificant ones. Extensive experiments show that DeCo achieves superior performance among pixel diffusion models, attaining FID of 1.62 (256x256) and 2.22 (512x512) on ImageNet, closing the gap with latent diffusion methods. Furthermore, our pretrained text-to-image model achieves a leading overall score of 0.86 on GenEval in system-level comparison. Codes are publicly available at https://github.com/Zehong-Ma/DeCo.

1. Introduction

DeCo addresses pixel diffusion’s difficulty in jointly modeling low-frequency semantics and high-frequency signals by assigning them to specialized modules. It further uses frequency-aware flow matching, achieving strong ImageNet and text-to-image results.

  • Pixel diffusion avoids VAE reconstruction limitations but must learn complex high-frequency signals and low-frequency semantics directly in the large pixel space.High-frequency noise can distract the DiT from learning low-frequency semantics, producing noisy outputs and degraded image quality.
  • DeCo assigns low-frequency semantic modeling to a DiT and high-frequency reconstruction to a lightweight pixel decoder conditioned on DiT semantics.The DiT uses downsampled inputs, while the decoder predicts pixel velocities from high-resolution inputs.
  • Frequency-aware flow matching transforms pixel velocities into the frequency domain and adaptively weights bands using JPEG quantization priors.The loss emphasizes visually salient frequencies while suppressing perceptually insignificant high-frequency components.
  • DeCo implicitly decouples frequencies through a pixel-decoder shortcut, unlike approaches that explicitly separate high-frequency noise and clean images.The authors identify this motivation as aligned with concurrent JiT while distinguishing their architectural solution.
  • FID reaches 1.62 at 256×256 and 2.22 at 512×512 on ImageNet, with DeCo reported as superior among pixel diffusion models and closer to latent diffusion.The paper also reports leading GenEval and DPG-Bench scores for its pretrained text-to-image model.

2. Related Work

Related work spans latent diffusion, pixel diffusion, and frequency-decoupled generation. DeCo differs by performing explicit multi-scale architectural frequency decoupling directly in pixel space with an efficient decoder.

  • Latent diffusion reduces spatial dimensionality by operating in VAE-learned latent space, but VAE artifacts and training complexity can limit the overall pipeline.Poorly trained VAEs may produce decoding artifacts that limit latent diffusion generalization.
  • Pixel diffusion avoids VAE dependence but progresses more slowly because raw pixel space has vast dimensionality and expensive multi-scale modeling.Prior methods use long residual connections or resolution-stage decomposition to reduce computational costs.
  • Earlier cascaded methods approximate temporal frequency decoupling but still use one architecture across frequencies and retain high-frequency noise.They also rely on complex denoising schedules and small patch sizes that reduce training or sampling efficiency.
  • DeCo performs explicit architectural frequency decoupling within each timestep, using a multi-scale pixel design for end-to-end pixel diffusion.Its decoder uses attention-free linear layers, making it more efficient for large-scale inputs than DDT’s attention-based DiT blocks.

3. Method

DeCo decouples low-frequency semantic modeling from high-frequency detail generation in pixel diffusion by pairing a downsampled-input DiT with a lightweight pixel decoder. It further guides optimization with frequency-aware flow matching and validates the separation through spectral and efficiency analyses.

  • 3.1. Overview: DeCo uses a DiT on downsampled inputs for low-frequency semantics and a lightweight pixel decoder on full-resolution inputs for high-frequency details.The decoder conditions detail generation on DiT semantics and predicts the final pixel velocity.
  • 3.2. Pixel Decoder: The pixel decoder is an attention-free stack of linear decoder blocks and projections designed for efficient high-frequency modeling.It forms dense queries from the full-resolution noised image and uses DiT features to modulate them through AdaLN.
  • 3.4. Empirical Analysis: DeCo suppresses high-frequency energy in DiT outputs while preserving it in pixel velocity, indicating that high-frequency components shift into the pixel decoder.The analysis uses DCT energy spectra computed across 10K images and all diffusion steps.
  • 3.3. Frequency-aware FM Loss: The frequency-aware flow-matching loss transforms velocities with block-wise 8×8 DCT and reweights frequency bands using JPEG-derived perceptual priors.The objective combines standard flow matching, frequency-aware flow matching, and REPA alignment.
  • 3.4. Empirical Analysis: 2.57 FID is reached in 400k iterations, reported as 10× faster than the baseline.The comparison is presented in Figure 5 as an FID-versus-training-progress analysis.

4. Experiments

DeCo is evaluated against pixel- and latent-diffusion baselines on class-to-image and text-to-image generation, with ablations examining its architecture and efficiency. It achieves strong image quality while reducing training and inference costs.

  • 4.1. Comparison with Baselines: Replacing the baseline’s final DiT blocks with the pixel decoder reduces FID from 61.10 to 34.12, while the frequency-aware loss further reduces it to 31.35.The frequency-decoupled architecture also raises IS from 16.81 to 46.44 before the additional loss improvement.
  • 4.2. Class-to-Image Generation: At 256×256, DeCo generates an image in 1.05s with 100 inference steps, compared with 38.4s for RDM and 9.78s for PixelFlow.Its single training iteration takes 0.24s, compared with 1.61s for PixelFlow.
  • 4.2. Class-to-Image Generation: DeCo reaches FID 1.90 after 320 epochs, versus 2.79 for the baseline, and achieves a 10× training-efficiency improvement.It reaches FID 2.57 in 80 epochs, exceeding the baseline’s FID at 800 epochs.
  • 4.2. Class-to-Image Generation: FID 1.62 at 256×256 and 2.22 at 512×512 establish DeCo as a leading pixel-diffusion method competitive with two-stage latent diffusion.These results are reported for ImageNet class-to-image generation.
  • 4.3. Text-to-Image Generation: 0.86 on GenEval gives DeCo leading text-to-image performance, while its DPG-Bench score remains comparable to two-stage latent diffusion.The comparison includes prominent text-to-image and unified models.
  • Ablation Studies: A hidden size of 32 and patch size 1 provide the best decoder ablation results, with the resulting attention-free decoder using 8.5M parameters.Larger hidden sizes provide no further gains, while patchifying the input degrades performance.

5. Conclusions

DeCo is a frequency-decoupled pixel-diffusion framework that separates low-frequency semantic modeling from high-frequency signal generation. It improves generation quality and efficiency, with leading pixel-diffusion results across class-to-image and text-to-image benchmarks.

  • 5. Conclusions: DeCo separates low-frequency semantic modeling in a DiT from high-frequency signal generation in a lightweight pixel decoder.This frequency decoupling is the framework’s central design.
  • 5. Conclusions: The frequency-aware flow-matching loss further improves visual quality by prioritizing perceptually important frequencies.
  • 5. Conclusions: DeCo achieves leading performance in pixel diffusion on class-to-image and text-to-image benchmarks, narrowing the gap with two-stage latent diffusion.

A. Comparison with JiT

The JiT comparison evaluates DeCo’s implicit frequency decoupling through its pixel-decoder shortcut. The comparison uses shared training and inference settings and reports validation against JiT-based alternatives.

  • A. Comparison with JiT: DeCo’s pixel decoder provides a high-frequency shortcut, while DiT guidance models low-frequency content in an end-to-end separation.The decoder directly models raw pixels and is guided by xlow.
  • A. Comparison with JiT: Visualizations, spectrum analysis, and Table 1 validate DeCo’s implicit frequency decoupling and its superiority over JiT+REPA.
  • A. Comparison with JiT: Baseline comparisons train ImageNet models at 256×256 for 200k iterations using a large DiT variant, with shared optimizer and architectural choices.
  • A. Comparison with JiT: Inference uses 50 Euler steps without classifier-free guidance for most models, while PixelFlow uses 100 steps and additional baselines include DiT-L/2, PixelFlow, PixNerd, and PixDDT.

B.2. Class-to-Image Generation

The class-to-image experiments train first at 256×256 and then fine-tune at 512×512. Evaluation uses guided Euler sampling with latency measured on a single A800 GPU.

  • B.2. Class-to-Image Generation: Class-to-image models are trained at 256×256 for 320 epochs and fine-tuned at 512×512 for 20 additional epochs.
  • B.2. Class-to-Image Generation: Inference uses 100 Euler steps with classifier-free guidance and a guidance interval.
  • B.2. Class-to-Image Generation: Inference latency is measured on a single A800 GPU, with batch size and learning rate following the default settings.

B.3. Text-to-Image Generation

The text-to-image model is trained on the BLIP3o dataset with Qwen3-1.7B as its text encoder. Training uses additional transformer-layer adaptation to improve alignment of frozen text features.

  • B.3. Text-to-Image Generation: Text-to-image training uses approximately 36M pretraining images and 60k high-quality instruction-tuning examples from BLIP3o.
  • B.3. Text-to-Image Generation: Qwen3-1.7B is used as the text encoder for the text-to-image model.
  • B.3. Text-to-Image Generation: Several transformer layers are jointly trained on frozen text features to improve their alignment, following a strategy similar to Fluid.
  • B.3. Text-to-Image Generation: The complete training run takes about 6 days on 8× H800 GPUs.

B.4. Experiment Configurations

The experiment configurations follow established diffusion-model training setups, while the frequency-aware loss uses JPEG-derived quantization tables to weight visually important components.

  • B.4. Experiment Configurations: Training follows setups from DiT, SiT, and PixNerd, with CFG scales swept at 0.1 intervals.The configurations cover DeCo-L/16, DeCo-XL/16, and DeCo-XXL/16.
  • B.4. Experiment Configurations: The tables reflect greater human sensitivity to low-frequency information and luminance than to high-frequency information and chrominance.An 8×8 table determines compression for each DCT frequency coefficient.
  • B.4. Experiment Configurations: JPEG quantization tables assign larger adaptive weights to visually important frequency components in the frequency-aware flow-matching loss.The weights use normalized reciprocals of scaled quantization tables.

E.1. Training Step of DeCo

DeCo trains a DiT to produce low-frequency semantic conditions and a lightweight pixel decoder to predict high-resolution pixel velocities. Its training combines standard flow matching with a JPEG-inspired frequency-aware loss and REPA loss, while accompanying procedures analyze spectral behavior and visualize outputs.

  • E.1. Training Step of DeCo: DeCo conditions a lightweight pixel decoder on DiT semantics to predict high-resolution pixel velocities, separating semantic modeling from detail generation.The DiT receives downsampled inputs, while the decoder uses high-resolution inputs and semantic conditions.
  • E.1. Training Step of DeCo: The training objective combines pixel-space flow-matching loss, frequency-aware flow-matching loss, and REPA loss.Frequency-aware loss compares DCT-transformed predicted and target velocities using normalized adaptive weights.
  • E.1. Training Step of DeCo: The frequency-aware loss emphasizes visually salient frequencies and suppresses perceptually insignificant high-frequency components.Its adaptive weights are derived from scaled JPEG quantization tables.
  • E.1. Training Step of DeCo: DCT spectral analysis applies 8×8 transforms, converts coefficients to energy, orders them by frequency, and log-normalizes the resulting measurements.Zigzag indexing maps two-dimensional coefficients from low to high frequency.
  • E.1. Training Step of DeCo: The supplementary visualizations include text-to-image results and class-to-image results at 256×256 and 512×512 resolutions.The text-to-image model supports Chinese, Japanese, and English with the Qwen3 text encoder after BLIP3o pretraining.
Loading 2511.19365v2…