PaperlayerView exact revision on arXiv

Source-linked AI summary

Energy-Guided Flow Matching

Haoyang Tong, Yu He, Fang Li, Lichen Ma, Jingling Fu, Dong Chen, Zhen Chen, Junshi Huang, Jie Cao

arXiv:2608.05811v3cs.CV

TL;DR

Pixel-space generation must coordinate global structure with fine-grained detail, while standard flow matching leaves spectral evolution implicit. EG-FM uses a sample-adaptive moving endpoint for coarse-to-fine generation and consistently improves quality and training efficiency, reaching an FID of 1.45 at 256×256.

  • Problem

    Pixel-space generation remains challenging because models must coordinate global structure with local high-frequency details, while standard flow matching leaves frequency evolution implicit.

  • Method

    EG-FM replaces the fixed clean endpoint with a heat-kernel-filtered, sample-adaptive endpoint that progressively releases spectral energy from low to high frequencies.

  • Results

    EG-FM consistently improves generation quality and training efficiency across class-conditional and text-to-image generation without architectural changes or additional losses, achieving an FID of 1.45 at 256×256.

  • Takeaways & Limitations

    The results support EG-FM as an effective trajectory-design approach for pixel-space image generation across class-conditional and text-to-image settings.

  • Takeaways & Limitations

    EG-FM has not yet been evaluated on heterogeneous or temporally extended signals or on the largest recent foundation backbones.

Abstract

from arXiv · show

Pixel-space generative models bypass lossy latent compression, yet necessitate joint learning of global structure and fine-grained details in a high-dimensional space. Standard flow matching interpolates noise toward a fixed clean-image endpoint, leaving the spectral evolution to be learned implicitly. In this paper, we introduce Energy-Guided Flow Matching(EG-FM) that explicitly models a coarse-to-fine generative trajectory by moving endpoint. Specifically, EG-FM replaces the fixed endpoint with a heat-kernel-filtered endpoint that evolves smoothly from low-frequency image to clean image. The fraction of high-frequency signal in moving endpoint is released by an image-specific energy-guided scheduling, leading to the re-targeting of velocity in flow matching. Our framework requires no adaptation of the backbone and training data, bringing negligible cost on the training and inference stages. In our experiment, EG-FM consistently achieves lower FID on the ImageNet class-conditional image generation task at $256 \times 256$ with fewer epochs, reaching an FID of 1.55 at 200 epochs and 1.45 at 600 epochs. We continue training the generation task on the setting of $512 \times 512$ resolution, yielding a FID of 1.58 after only 40 high-resolution adaptation epochs. Furthermore, we transfer EG-FM on text-to-image generation and achieve 0.85 on GenEval score and 83.9 on DPG-Bench. Code is available at https://github.com/ysng123/EG-FM.

1 Introduction

Pixel-space generation preserves fine-grained detail but remains difficult because flow matching does not explicitly model coarse-to-fine frequency evolution. EG-FM addresses this by replacing the fixed endpoint with a sample-adaptive moving endpoint controlled by image spectral energy.

  • Motivation: Pixel-space models avoid latent-compression information loss and preserve fine-grained image details, but high-dimensional generation remains challenging.These models learn image distributions directly on pixels.
  • Problem: Standard flow matching connects noise to a fixed clean-image endpoint, assuming the same target throughout generation.This formulation does not explicitly represent how visual information is recovered over time.
  • Problem: Low-frequency components establish global structure early, whereas high-frequency textures and details emerge later in pixel-level generation.An explicit frequency-aware path could provide an inductive bias for this coarse-to-fine progression.
  • Method: EG-FM replaces the fixed clean endpoint with a moving endpoint that evolves according to each image’s frequency composition.The method constructs smooth heat-time scheduling and controls information release using scheduled energy.
  • Method: EG-FM progressively releases frequency information according to spectral energy and derives the corresponding velocity target for compatibility with standard flow matching.Its unified energy-based schedule is sample-adaptive.

2 Related Work

Related work addresses the difficulty of high-resolution pixel-space generation through resolution-aware optimization and representation or process decompositions. In contrast, EG-FM directly embeds sample-adaptive frequency release into the probability path to supervise coarse-to-fine generation.

  • High-resolution pixel-space generation: High-resolution pixel-space generation is difficult because dense RGB representations increase optimization complexity and computational cost.
  • High-resolution pixel-space generation: Existing methods mainly improve high-resolution optimization through resolution-aware noise scheduling and related strategies, while distributing computation across resolutions.
  • Frequency-guided generation: Most existing methods optimize along a predefined probability path, leaving temporal spatial-frequency emergence to be inferred implicitly from training data.
  • Frequency-guided generation: EG-FM embeds sample-adaptive frequency release directly into the probability path and explicitly supervises coarse-to-fine ordering.
  • Coarse-to-fine generative priors: Coarse-to-fine priors decompose representations or generative processes across branches, scales, transport paths, or auxiliary paths, using frequency-aware structures and objectives.

3 Method

Energy-Guided Flow Matching models generation as a coarse-to-fine trajectory with moving spectral endpoints, progressively releasing frequency components from low-pass structure to the clean image. It uses image-adaptive heat-time scheduling and an exact moving-endpoint velocity target while preserving standard flow-matching denoising progress.

  • 3.1 Energy-Guided Flow Matching: EG-FM integrates a coarse-to-fine prior into the generation endpoints, guiding intermediate states from global structure toward local details.The framework comprises moving endpoints, sample-adaptive heat-time scheduling, and a target-velocity derivation.
  • 3.1 Energy-Guided Flow Matching: Moving endpoints evolve continuously from a low-pass image toward the full-spectrum image by progressively releasing high-frequency components.The endpoint satisfies y1(x) = x, preserving the clean-image boundary at the final time.
  • 3.2 Sample-Adaptive Heat-Time Scheduling: Heat-kernel responses generate a smooth, nested family of low-pass images whose degradation strength is controlled by the image-specific monotonic heat-time h(x, t).The heat-kernel acts as a low-pass filter initially and releases more high-frequency signal as time approaches 1.
  • 3.2 Sample-Adaptive Heat-Time Scheduling: The information gap measures missing spectral energy, enabling sample-adaptive scheduling that aligns the same time t with comparable fractions of energy recovery across images.A nondecreasing global release clock q(t) satisfies q(0) = 0 and q(1) = 1; h(x, t) is recovered numerically by inverting the information-gap function.
  • 3.3 Target Velocity of Moving Endpoint: Because the endpoint moves, EG-FM trains on the exact derivative of the noisy-sample path while leaving standard flow-matching denoising progress unchanged.Only the path and target change relative to fixed-endpoint flow matching.

4 Experiments

EG-FM improves ImageNet class-conditional generation across backbones and resolutions, converges faster under matched sampling protocols, and transfers to text-to-image generation. Ablations attribute its gains to sample-adaptive spectral scheduling, a quintic smootherstep release clock, and effectiveness across model scales.

  • ImageNet 256×256: EG-FM improves ImageNet 256×256 FID across DeCo, HyperDiT, and PixelDiT while accelerating convergence under fixed backbones, samplers, and evaluation protocols.DeCo-XL/16 reaches FID 1.63 after 440 epochs versus 1.69 after 600 baseline epochs; HyperDiT-H reaches 1.51 after 220 versus 1.56 after 600.
  • ImageNet 256×256: 1.45 FID is achieved by PixelDiT-XL at 600 epochs, after reaching 1.55 at 200 epochs versus 1.61 for the 320-epoch baseline.The controlled comparison changes the probability trajectory while holding the backbone architecture, sampler, and evaluation protocol fixed.
  • ImageNet 512×512: 1.58 FID is achieved by HyperDiT with EG-FM at 512×512 after limited high-resolution adaptation, while PixelDiT reaches 1.68 FID and 295.5 Inception Score.PixelDiT EG-FM is compared with standard PixelDiT at 1.81 FID and 278.6 Inception Score after 530 additional epochs.
  • Text-to-image: EG-FM transfers beyond class conditioning to text-to-image generation, evaluating text-conditioned semantic preservation with complementary GenEval and DPG-Bench metrics.GenEval emphasizes object-centric composition, whereas DPG-Bench stresses dense-prompt compliance.
  • Ablations: Sample-level spectral adaptation performs best among the compared heat-time schedules, and the quintic smootherstep gives the best FID among the tested release-clock functions.EG-FM consistently outperforms Standard-FM across three model scales; CFG tuning yields a minimum FID of 1.45 at CFG = 2.55.

5 Conclusion … A.3 Implicit Derivative and Exact Velocity

EG-FM uses a sample-adaptive, heat-kernel-filtered moving endpoint to impose coarse-to-fine spectral generation, while preserving boundary distributions and improving generation quality and training efficiency. Its derivation defines an energy-matched heat-time schedule with a unique recoverable solution and an exact velocity target for training.

  • 5 Conclusion: EG-FM replaces the fixed clean endpoint with a heat-kernel-filtered moving endpoint and derives sample-adaptive heat time from recovered spectral energy.The schedule governs the same frequency-release rate across images.
  • 5 Conclusion: EG-FM consistently improves generation quality and training efficiency for class-conditional and text-to-image generation without architectural changes or additional losses.The conclusion presents these results as validation of EG-FM for pixel-space image generation.
  • A Detailed Derivation of EG-FM: The derivation establishes the full sample-adaptive heat-time schedule and its path-consistent velocity target.This appendix section supplies the formal basis for the schedule and velocity construction.
  • A.1 Spectral Endpoint Path: The moving spectral endpoint exposes coarse-to-fine ordering while retaining the original boundary distributions z0 = ϵ and z1 = x.The initial endpoint is a low-pass image, the final endpoint is the clean image, and only intermediate transport geometry changes.
  • A.2 Energy-Matched Heat Time: The sample-dependent heat time matches each path time to the same fraction of recovered spectral change across samples.Absolute heat time can remain sample-dependent even when the recovered-energy fraction is aligned.
  • A.2 Energy-Matched Heat Time: For a nondegenerate image spectrum, the heat-time equation has a unique solution in [0, 1] and can be recovered reliably by bracketed bisection.The strict interior inequality together with the boundary conditions establishes uniqueness.
  • A.3 Implicit Derivative and Exact Velocity: The implicit derivative satisfies ∂th(x, t) ≤0, ensuring progressively weakening attenuation, and the exact differentiated state path provides the velocity regression target.If the release clock has zero derivative at both boundaries, endpoint motion vanishes smoothly at both ends.

A.4 Endpoint Stability · B Extension to x-Prediction · B.1 Path-Consistent Parameterization

The appendix establishes that EG-FM remains analytically and numerically stable at both endpoints, with bounded states and velocities and no probability-flow ODE singularity. It also derives a path-consistent conversion for clean-image (x-) prediction that accounts for spectral transformation and endpoint motion.

  • A.4 Endpoint Stability: Under the endpoint-flatness conditions, yt(x) and vt extend continuously to both endpoints, while ∂tyt(x) tends to zero as t approaches 0 or 1.These conditions are formulated for the general release clock and do not require a particular functional form.
  • A.4 Endpoint Stability: The apparent 0/0 endpoint expression has a finite, vanishing limit, implying ∂tyt(x) → 0 near t = 0.The result follows from the first endpoint-flatness condition together with Eq. (31).
  • A.4 Endpoint Stability: At t → 1, the second endpoint-flatness condition likewise gives ∂tyt(x) → 0 and proves the stated endpoint assignments.Both the state and velocity remain bounded in endpoint neighborhoods, so the probability-flow ODE has no endpoint singularity induced by the energy-matched schedule.
  • A.4 Endpoint Stability: For q(t) = 6t5 −15t4 + 10t3, endpoint-flatness can be checked without evaluating ratios of small quantities.The explicit endpoint rates also show that the endpoint-motion contribution t ∂tyt(x) vanishes at both ends.
  • A.4 Endpoint Stability: In implementation, endpoint regions t ≤ 10−5 and 1 − t ≤ 10−5 use direct endpoint expressions, while other times use the general derivative expression.The analytic assignments are (h, ∂th) = (1, 0) at t = 0 and (h, ∂th) = (0, 0) at t = 1; scalar spectral reductions, root finding, and derivatives use FP32.
  • B Extension to x-Prediction: EG-FM supports backbones that predict the clean image by defining an energy-guided endpoint operator from the prediction’s spectrum using the same energy-matching rule.The endpoint implied by the clean-image prediction follows from this operator.
  • B.1 Path-Consistent Parameterization: The clean-image parameterization computes endpoint motion analytically from the path-time dependence, without differentiating the neural network with respect to time.The derivation holds the predicted clean image fixed while differentiating the explicit path dependence.
  • B.1 Path-Consistent Parameterization: Equation (46) gives the path-consistent conversion from x-prediction to velocity prediction, including spectral transformation and endpoint-motion terms.Using the usual fixed-endpoint conversion would be inconsistent with EG-FM training states; at the terminal boundary, the clean prediction is used directly.

B.2 JiT Result and Discussion · C Experimental Details

EG-FM transfers to JiT’s x-prediction parameterization, improving FID from 2.37 to 2.33 under the same backbone and evaluation setting. The smaller gain is attributed to unreliable early-time spectral predictions when global structure is first established.

  • B.2 JiT Result and Discussion: 2.33 FID is achieved with EG-FM versus 2.37 for the controlled JiT x-prediction experiment.The comparison uses the same backbone and evaluation setting.
  • B.2 JiT Result and Discussion: EG-FM’s path transfers to JiT’s x-prediction parameterization.The result demonstrates compatibility beyond direct velocity prediction.
  • B.2 JiT Result and Discussion: The gain is positive but smaller than gains observed with direct velocity prediction.The passage links this difference to information available when heat time is evaluated.
  • B.2 JiT Result and Discussion: Predictions at early solver times remain noisy and spectrally inaccurate.This occurs when global structure is first being established.
  • B.2 JiT Result and Discussion: Consequently, Eex(ρ) can be unreliable at early solver times.The passage identifies noisy and spectrally inaccurate predictions as the cause.
  • B.2 JiT Result and Discussion: The inferred h(ex, t) can likewise be unreliable when global structure is first established.This unreliability helps explain the smaller improvement under JiT’s x-prediction parameterization.

C.1 Detailed Results on ImageNet 512×512

Table 7 broadens the 512×512 ImageNet comparison beyond FID and Inception Score by adding sFID, precision, recall, and sampling cost. These metrics assess spatial fidelity, sample quality, distributional coverage, and computational expense.

  • Evaluation Metrics: Table 7 compares high-resolution results using sFID, precision, recall, and sampling cost alongside FID and Inception Score.The expanded evaluation covers both generation quality and sampling cost.
  • Evaluation Metrics: sFID, precision, recall, and Inception Score measure spatial fidelity, sample quality, and distributional coverage, while sampling cost captures computational expense.The passage explicitly groups these metrics by the aspects they evaluate.

C.2 Detailed Text-to-Image Results

The detailed text-to-image evaluation decomposes aggregate scores into capability-specific criteria. GenEval focuses on visual composition abilities, while DPG-Bench emphasizes dense-prompt understanding.

  • GenEval: GenEval evaluates object rendering, counting, color, spatial relations, and attribute binding.These criteria decompose text-to-image performance into distinct visual composition capabilities.
  • DPG-Bench: DPG-Bench evaluates global consistency, entities, attributes, relations, and other dense-prompt requirements.Its criteria target consistency and multi-entity prompt adherence.

C.3 Release-Clock Ablation

The release-clock ablation compares four monotone schedules that recover the same total spectral gap but differ in temporal allocation. Quintic smootherstep performs best, delaying early recovery while concentrating it near the path’s middle and maintaining zero endpoint first derivatives.

  • Four monotone release clocks are compared in the ablation.
  • Normalization enforces q(0) = 0 and q(1) = 1, so clocks differ only in temporal allocation.
  • 1.99 FID: quintic smootherstep achieves the best 80-epoch result.
  • Linear release is uniform, whereas nonlinear clocks delay early recovery and concentrate it near the path’s middle.
  • Smootherstep retains zero first derivatives at both endpoints, consistent with the stability analysis.

C.4 Training and Inference Efficiency · C.5 EG-FM Algorithms · C.6 FID Evaluator Consistency

EG-FM adds negligible training overhead while preserving Standard FM inference cost through direct velocity prediction. Its training targets use a sample-adaptive heat-time solver, and FID evaluation remains consistent across torch-fidelity and ADM implementations.

  • C.4 Training and Inference Efficiency: 0.0256% is the largest measured per-sample compute increase from EG-FM target construction relative to backbone computation.Across matched runs, per-step wall-clock changes range from +0.41% to +4.81%, with all configurations except PixelDiT-B/16 within 1% of Standard FM.
  • C.4 Training and Inference Efficiency: EG-FM does not change inference FLOPs or wall time because it uses the same backbone, solver, time grid, and number of function evaluations as Standard FM.Inference directly evaluates the learned velocity field without filtering, FFT, inverse FFT, energy computation, or bisection.
  • C.5 EG-FM Algorithms: Algorithm 1 computes the sample-adaptive heat time ht = h(x, t) and derivative ˙ht with bracketed bisection and implicit differentiation for t ∈(0, 1) and eGx > 0.The solver initializes a bracket and iteratively updates it using filtered-energy comparisons before returning ht and its derivative.
  • C.5 EG-FM Algorithms: Both EG-FM algorithms form training targets only, while inference integrates dzt/dt = vθ(zt, t, c) directly from Gaussian noise.This inference protocol follows the direct velocity-prediction procedure described for the matched experiments.
  • C.5 EG-FM Algorithms: Table 7 evaluates 512×512 class-conditional ImageNet generation on 50K generated samples with the ADM evaluation suite, with NFE including conditional and unconditional classifier-free-guidance evaluations.The table marks continued training from a 256×256 checkpoint with † and uses dashes for unreported items.
  • C.6 FID Evaluator Consistency: FID drops sharply between 80 and 200 epochs and then improves steadily through 600 epochs under both torch-fidelity and the ADM evaluation suite.The two implementations show closely aligned convergence trends, while torch-fidelity can produce slightly lower absolute values than ADM.
  • C.5 EG-FM Algorithms: The EG-FM training procedure samples t and Gaussian noise, computes heat time and its derivative, constructs filtered endpoints, and trains against a re-targeted velocity.The listed procedure applies Fourier-domain filtering before forming zt and the target velocity v⋆.

D Limitations and Future Work … E.2 Text-to-Image Generation

The paper identifies untested modalities and model scales as limitations, while additional visualizations demonstrate EG-FM’s class-conditional and text-to-image outputs across resolutions, subjects, styles, and spatial relationships.

  • D Limitations and Future Work: EG-FM remains unevaluated for joint text-image modeling, video generation, embodied decision making, and the largest Flux- and Qwen-Image-scale backbones.Future work targets endpoint-guided flow matching across these modalities and scales.
  • E Additional Visualizations: Additional samples use the reported checkpoints and evaluation sampling configurations.
  • E.1 Class-Conditional Generation: Additional PixelDiT-XL samples at 256^2 and 512^2 span diverse categories and layouts while preserving global class structure and detailed textures.Each grid places two 512^2 samples at the top and 256^2 samples below.
  • E.2 Text-to-Image Generation: Additional 512^2 text-to-image results cover subjects, attributes, artistic styles, counting, and spatial relations.Prompt-aligned samples illustrate compatibility with text conditioning and recovery of fine visual details.
  • E.2 Text-to-Image Generation: EG-FM-T2I generates 512×512 samples spanning architectural scenes, weather, interiors, animals, and human-centered compositions.
  • E.2 Text-to-Image Generation: Representative GenEval samples test prompts involving object counts, relative positions, and object-color combinations.Examples include three buses, a dog right of a teddy bear, and colored objects paired with sports balls.
Loading 2608.05811v3…