Source-linked AI summary
Balancing Frequencies and Pixels in Flow Matching
Lucas Degeorge, Paul Couairon, Arijit Ghosh, Alexei A. Efros, David Picard, Vicky Kalogeiton
TL;DR
Pixel-space flow objectives overemphasize low-frequency residuals, delaying learning of perceptually important fine details. The paper introduces Focal Log-Frequency Loss and a staged frequency-to-pixel objective, reporting faster convergence and improved generation quality without architectural changes.
Problem
Pixel-space objectives are spectrally imbalanced: low-frequency components dominate optimization even though high-frequency details strongly affect visual quality.
Method
The paper introduces Focal Log-Frequency Loss and combines early frequency supervision with later standard pixel v-loss through a scheduled objective.
Results
The method accelerates convergence and consistently improves generation quality across model scales without architectural modifications.
Takeaways & Limitations
Frequency supervision is most useful early, while pixel supervision supports later spatial refinement, motivating their staged combination.
Takeaways & Limitations
The origin of spectral bias and the reason frequency-only training plateaus while later pixel losses perform better remain open questions.
Abstract
from arXiv · showhide
Natural images follow a $1/f^2$ spectral distribution: most signal energy lies in the low spatial frequencies, while the perceptually important structures such as textures and edges occupy sparse high-frequency bands. Pixel-space reconstruction objectives, however, treat all spatial errors uniformly, causing low frequencies to dominate the optimization signal and delaying the learning of fine-scale details. In this work, we identify this objective-level spectral imbalance as a key inefficiency in training pixel-space flow models. To address it, we propose a Focal Log-Frequency Loss (f-loss), a spectrally balanced objective that equalizes the learning signal across frequencies, emphasizing high-frequency components that are otherwise underrepresented in pixel-space objectives. Building on this, we introduce a simple training strategy that combines frequency and pixel supervision: we first emphasize frequency-domain learning early to capture all frequencies, and then transition to standard pixel-space v-loss for spatial refinement. This balancing mitigates the low-frequency bias of pixel losses and aligns the training signal with the evolving needs of the model. Our approach is conceptually simple, requires no architectural changes, and acts as a drop-in replacement for flow matching losses. Across multiple model scales, it accelerates convergence by up to 40% while consistently improving FID and perceptual fidelity. We will release code and models.
1 Introduction
Natural images concentrate signal energy in low frequencies while perceptually important edges and textures occupy sparse high-frequency bands. The paper addresses this mismatch with a spectrally balanced frequency loss and a frequency-to-pixel training strategy.
- Motivation: Natural images approximately follow a 1/f^2 spectrum, concentrating signal energy in low frequencies while edges and textures remain perceptually important high-frequency details.Low-frequency components describe global structure, whereas high-frequency components contain sparse but structured details.
- Method: The Focal Log-Frequency Loss applies focal weighting to logarithmically compressed spectral errors, increasing the influence of underrepresented high-frequency components.The objective is designed to equalize the training signal across the spectrum.
- Method: The proposed strategy begins with frequency-domain supervision to learn spectral structure, then uses standard pixel v-loss to refine spatial consistency.This combines early frequency learning with later pixel-space refinement.
- Motivation: Pixel-space objectives bias optimization toward low-frequency components despite the perceptual importance of high-frequency details.The paper identifies this mismatch between natural-image spectra and standard training objectives as a source of inefficiency.
- Results: The approach accelerates convergence and improves generation quality across model scales without requiring architectural modifications.It is presented as a drop-in replacement for standard flow-matching objectives and improves both quantitative FID and qualitative generation quality.
2 Related Work
Pixel-space flow and denoising models use spatial-domain regression objectives, while spectral bias makes high-frequency detail difficult to learn. Prior remedies often modify architectures or training procedures; this work instead rebalances the training objective without architectural changes.
- Flow Matching and Pixel-Space Synthesis: Diffusion and flow matching models typically predict noise or velocity through spatial-domain regression objectives.These objectives perform well in compressed latent spaces, but the passage frames pixel-space models as the focus of this work.
- Flow Matching and Pixel-Space Synthesis: The proposed frequency rebalancing loss targets optimization dynamics directly, improving convergence and performance without architectural modifications or auxiliary losses.The work presents this objective-level intervention as its central contribution for pixel-space models.
- Spectral Bias and Frequency-Aware Generative Modeling: Neural networks learn low-frequency structure faster than fine-grained high-frequency details, producing sharp-texture preservation challenges and high-frequency artifacts.The passage describes this spectral bias across generative paradigms.
- Spectral Bias and Frequency-Aware Generative Modeling: Existing frequency-balancing approaches include multi-scale strategies, dedicated frequency branches, and architectural frequency decoupling.These are presented as structural modifications for addressing frequency imbalance.
- Spectral Bias and Frequency-Aware Generative Modeling: Prior methods also rely on complex denoising schedules, post-hoc adjustments, or architectural changes.The authors distinguish their approach from these more complex interventions.
3 Method
The method diagnoses low-frequency bias in pixel-space flow matching and introduces frequency-aware supervision to balance spectral learning. It combines early f-loss emphasis with later v-loss refinement, improving convergence across model sizes without architectural changes.
- 3.1 Background: Flow matching learns a velocity field transporting noise to data, with linear interpolation yielding the ground-truth velocity v* = x1 − ε.The model predicts vθ(xt, t), while x_t = (1 − t)ε + t x1.
- 3.2 Spectral Diagnosis of Pixel-Space Flow Models: Pixel-space v-loss overestimates frequencies below 10^-1 cycles/pixel by 20% and underestimates frequencies above 8 × 10^-1 cycles/pixel by nearly -60%.The high-frequency deficit grows monotonically toward the Nyquist limit.
- 3.2 Spectral Diagnosis of Pixel-Space Flow Models: A two-frequency toy experiment shows that v-loss rapidly learns the low-frequency pattern while struggling to learn the high-frequency pattern.This supports an implicit optimization preference for low frequencies and slower learning of fine textures.
- 3.3 A Joint Pixel-Frequency Objective: The proposed f-loss reweights Fourier-domain reconstruction errors to equalize learning across frequencies and amplify underrepresented high-frequency components.It uses per-frequency residuals, adaptive focal normalization, and logarithmic compression to prevent any frequency from monopolizing the objective.
- 3.3 A Joint Pixel-Frequency Objective: The joint fv-loss schedules supervision from frequency space to pixel space because frequency loss is more effective early, whereas v-loss becomes more effective later.The schedule sets wf = λ(s) and wv = 1 − λ(s), with λ(s) decaying from 1 to 0 around the observed crossover point.
- 3.4 Empirical Validation: fv-loss outperforms v-loss across JiT-B/16, JiT-L/16, and JiT-XL/16 in both unguided and guided FID evaluations, with the largest gap in early epochs.Qualitative comparisons also show finer textures earlier with f-loss-based training, including roof hay and mitten knitting patterns.
4 Experiments
Experiments compare f-loss, v-loss, and the scheduled fv-loss across resolutions, model scales, architectures, and training budgets. The scheduled objective combines early frequency-domain advantages with later pixel-space refinement, improving convergence and generation quality while adding computational overhead.
- Experimental setup: FID evaluations use ImageNet class-conditional generation at 256×256 and 512×512, primarily with pixel-space flow matching following JiT.Models use standard FID, with FDD and IS additionally used for ablations.
- State-of-the-art comparison: fv-loss improves the standard JiT XL/16 result from 2.21 to 2.13 FID after 750K steps under identical architecture and training settings.With perceptual losses, it reaches FID 1.83 in 500k steps versus 800k for PixelGen, while improving IS.
- Ablation study: At 256×256, f-loss outperforms v-loss early in both unguided and guided settings, while the scheduled fv-loss improves both early and late training.For the B-size model, early unguided FID is 42.34 versus 46.02, and guided FID is 6.82 versus 8.44.
- Ablation study: At 512×512, f-loss is better early, whereas fv-loss becomes much better than either standalone loss as training progresses.Early FID comparisons include 9.48 versus 10.40 for B and 4.65 versus 6.41 for L at 80 epochs.
- Adaptive weighting: A sigmoid weighting schedule performs best throughout training, while reversing the schedule is among the worst configurations.The ablation supports emphasizing spectral balancing early and pixel supervision later.
- Generalization across architectures: 2× faster: fv-loss reaches the same FID as v-loss when training PixelDiT, and remains beneficial alongside REPA alignment loss.This tests generalization beyond the JiT architecture.
- Wall-clock speedup: fv-loss adds 14% training-time overhead over v-loss but overtakes it by 200 minutes, with the speedup compensating for cost from 300 minutes onward.For JiT-B trained for 400k steps, v-loss takes 432.0 minutes, f-loss 453.1 minutes, and fv-loss 492.3 minutes.
- Convergence dynamics: fv-loss consistently outperforms v-loss in guided FID and reduces average low- and high-frequency magnitude errors during training.Figure 6 reports the comparison against wall-clock time and frequency-band error.
5 Conclusion
The paper attributes early convergence inefficiency in pixel-space flow matching to low-frequency spectral bias and proposes scheduled frequency rebalancing. The resulting fv-loss accelerates convergence and achieves state-of-the-art pixel-space generation on class-conditional ImageNet.
- Conclusion: Pixel-space velocity loss overestimates low frequencies early, underestimates high frequencies, and slows early convergence because natural-image spectra decay as 1/f^2.The paper addresses this imbalance with a frequency-domain loss that equalizes frequencies.
- Conclusion: The fv-loss schedule combines the early speed advantage of frequency loss with the later performance of classical pixel-space velocity loss.The resulting method significantly accelerates flow-matching convergence.
- Limitations: The origin of spectral bias remains unresolved, including why frequency loss alone plateaus and why switching to pixel-space loss helps later.These questions are identified as directions for future work.
A Implementation Details
Implementation follows public JiT codebases and uses specified hardware configurations. The switch point is selected by intersecting validation FID curves rather than by grid search.
- Implementation details: Experiments closely follow public JiT codebases, with configurations detailed in Table 5.Most experiments use four nodes of four H100 GPUs; final XL models use four nodes of four GB200 GPUs.
- Switch-point selection: The switch point s⋆ is chosen where f-loss and v-loss FID curves intersect on a held-out validation set.The crossing occurs around 200k steps for B and L models and around 350k steps for XL models.
B Additional Results
The ablations show that logarithmic residual compression and normalized spectral weighting are central to f-loss, while the frequency-to-pixel schedule remains beneficial across alternative frequency objectives.
- f-loss components: Logarithmic compression of spectral residuals substantially improves f-loss over the original focal frequency loss.The transformation uses log(1 + e_u,v) before reweighting, reducing domination by the largest residuals and stabilizing spectral balancing.
- f-loss components: Normalized weighting preserves relative frequency difficulty, whereas unnormalized weights become dominated by a subset of frequencies.The ablation reports that combining logarithmic compression with normalization yields the best performance.
- f-loss components: The combined design achieves an FID of 41.78 (cfg=1.0) and 6.39 (cfg=3.5).
- Frequency-to-pixel schedule: The frequency-to-pixel schedule improves results across alternative frequency-domain losses, not only f-loss.The comparison uses JiT-B at 256^2 resolution with discrete focal frequency, the original loss, and a Laplacian pyramid loss.
B.2 Frequency-Domain Error Analysis
Frequency-domain supervision reduces spectral error earlier than pixel-space v-loss, while combining it with late pixel refinement addresses f-loss’s phase insensitivity. The reported analysis also finds only partial transfer to latent-space training.
- Frequency-domain error dynamics: At 50k–75k steps, f-loss has lower magnitude error than v-loss in both low- and high-frequency bands.The analysis interprets this as faster correction of spectral bias during early training.
- Frequency-domain error dynamics: f-loss equalizes frequency learning but is insensitive to phase, whereas v-loss enforces local spatial coherence and phase alignment.The distinction motivates using frequency supervision for early spectral alignment and pixel supervision for precise spatial refinement.
- Latent-space generalization: In latent-space SiT training, f-loss has lower FID than v-loss at epoch 80, but neither matches the original SiT model’s FID 33.Both compared models use x-prediction, while the original SiT model uses v-prediction.
- Latent-space generalization: The spectral-bias mechanism studied is predominantly a pixel-space phenomenon, leaving its manifestation in latent space unclear.The VAE is described as discarding much of the high-frequency detail present in pixel space.
- Toy experiment: The toy experiment trains an MLP on synthetic images containing exactly one low- and one high-frequency component.The experiment isolates the temporal dynamics of spectral bias and reports Fourier-spectrum evolution.
D Focal Frequency Loss as a Continuous Laplacian Pyramid Loss
The paper frames f-loss as a continuous frequency-domain generalization of Laplacian-pyramid decomposition, using logarithmic spectral residuals to balance scale-specific structure. Fine-grained continuous weighting outperforms coarser discrete alternatives in the reported comparison.
- Continuous Laplacian-pyramid interpretation: f-loss generalizes octave-based Laplacian-pyramid decomposition into a continuous, infinite-band frequency representation.The method maps images into the frequency domain with the DFT rather than using explicit hierarchical branching.
- Continuous Laplacian-pyramid interpretation: The spectral difference e_u,v = ||F_θ(u, v) − F(u, v)||_2 serves as the functional equivalent of a Laplacian residual.The logarithmic mapping gives equal structural weight to each frequency doubling.
- Continuous Laplacian-pyramid interpretation: The log-transformed residual prevents high-energy low-frequency components from dominating the loss.This transfers the scale-balancing rationale of Laplacian pyramids to the full Fourier spectrum.
- Empirical study: The discrete-versus-continuous comparison is reported in Table 9, including unguided Precision, Recall, Density, and Coverage.
- Empirical study: Increasing the number of discrete spectral cells improves performance, while the continuous formulation achieves FID 41.78.The comparison evaluates frequency bands m = 2, 3, 4 and orientations k = 1, 4.
E Extended comparison in early stages of training
The extended early-training comparison examines f-loss and v-loss quantitatively during their initial convergence stages.
- Extended early-stage comparison: Figure 9 extends the quantitative comparison between f-loss and v-loss during early training.
F Additional qualitative results at 512 × 512 resolution
The section presents additional qualitative results at 512 × 512 resolution and identifies the associated training setup. It also extends the early-training comparison between f-loss and v-loss.
- Additional qualitative results show 512 × 512 images generated by the model trained with fv-loss.The model was trained for 400k steps.
- The extended comparison examines f-loss and v-loss during the early stages of training.