PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evaluation Unlocks Any-Step Text-to-Image Generation

Xin Yu, Xiaojuan Qi, Zhengqi Li, Kai Zhang, Richard Zhang, Zhe Lin, Eli Shechtman, Tianyu Wang, Yotam Nitzan

arXiv:2512.22374v1cs.CVcs.AIcs.LG

TL;DR

Text-to-image models trained with local supervision typically need many inference steps, while few-step distillation relies on a pretrained teacher. Self-E combines learning from data with self-evaluation of generated samples, and performs strongly at few steps while scaling with more steps.

  • Problem

    Local supervision in diffusion and flow models typically requires many sequential inference steps, while distillation-based few-step methods rely on pretrained teachers.

    Source: §1 · p. 1

  • Method

    Self-E trains from scratch using data-derived local velocity supervision and self-evaluation of generated samples with current score estimates as a dynamic self-teacher.

    Source: §1 · pp. 1–2

  • Results

    Self-E surpasses diffusion, flow, distillation-based, and concurrent any-step models below 8 steps, is competitive with or surpasses some state-of-the-art flow methods at 50 steps, and improves monotonically as steps increase.

    Source: §1 · p. 2

  • Takeaways & Limitations

    One model supports both few-step generation and high-quality long-trajectory sampling as inference steps increase.

    Source: §1 · p. 2

    At extremely low counts such as 1–2 steps, image details may lack sharpness compared with 50-step inference.

    Source: §S.5 · p. 17

Abstract

from arXiv · show

We introduce the Self-Evaluating Model (Self-E), a novel, from-scratch training approach for text-to-image generation that supports any-step inference. Self-E learns from data similarly to a Flow Matching model, while simultaneously employing a novel self-evaluation mechanism: it evaluates its own generated samples using its current score estimates, effectively serving as a dynamic self-teacher. Unlike traditional diffusion or flow models, it does not rely solely on local supervision, which typically necessitates many inference steps. Unlike distillation-based approaches, it does not require a pretrained teacher. This combination of instantaneous local learning and self-driven global matching bridges the gap between the two paradigms, enabling the training of a high-quality text-to-image model from scratch that excels even at very low step counts. Extensive experiments on large-scale text-to-image benchmarks show that Self-E not only excels in few-step generation, but is also competitive with state-of-the-art Flow Matching models at 50 steps. We further find that its performance improves monotonically as inference steps increase, enabling both ultra-fast few-step generation and high-quality long-trajectory sampling within a single unified model. To our knowledge, Self-E is the first from-scratch, any-step text-to-image model, offering a unified framework for efficient and scalable generation.

1. Introduction

The introduction frames the need for text-to-image generation with fewer inference steps and presents Self-E as a from-scratch approach combining local learning with self-evaluation. It reports strong few-step results and scaling across inference budgets.

1. Introduction

  • Local score or velocity supervision gives short-range guidance, so diffusion and flow models typically need dozens of sequential steps, making inference costly for time-sensitive uses.Source: §1 · p. 1
  • Distillation enables few-step inference through global teacher-matching objectives, while from-scratch consistency methods can be unstable or lose quality and large text-to-image systems still rely heavily on distillation.The introduction describes consistency methods as learning flow maps or average velocity for a one-step shortcut [15] [22] [30] [59].Source: §1 · p. 1
  • Local score estimates evaluate generated samples, providing a dynamic self-teacher alongside data-derived local velocity supervision.The approach combines instantaneous local learning with self-driven global matching, without a pretrained teacher.Source: §1 · pp. 1–2
  • Figure 1 presents diverse outputs at different inference step counts, described as showing coherent semantics and strong text alignment.Source: Figure 1 · p. 2
  • Self-E surpasses diffusion, flow, distillation-based, and concurrent any-step models below 8 steps, and is competitive with or surpasses some state-of-the-art flow methods at 50 steps.Its performance improves monotonically with more inference steps.Source: §1 · p. 2

2. Background: Flow Matching

The section describes flow matching as learning a velocity field along a noisy trajectory, then explains why local velocity estimates may be inadequate for one-step generation. The accompanying model overview distinguishes learning from data and self-evaluation.

2. Background: Flow Matching

  • Flow matching and diffusion learn the instantaneous velocity field, or equivalently the score, induced by a continuous-time forward diffusion process.A noise scheduler defines the noisy distribution, and conditional flow matching trains a network to predict marginal velocity using mean-squared error.Source: §2 · p. 2
  • The conditional flow-matching loss minimizes mean-squared error between predicted and conditional velocities.The equation passage identifies this as the loss minimized by θ* in Eq. (3).Source: §2 · pp. 2–3
  • Because velocity is local, one step captures only an immediate direction and may recover an average of possible original samples rather than account for trajectory curvature.Source: §2 · pp. 2–3
  • Figure 2 presents Self-E as combining learning from data and self-evaluation; the latter uses stop-gradient passes on re-noised generated samples to derive feedback for global distribution matching.Source: Figure 2 · p. 3, §2 · p. 3

3. Self-Evaluating Model

Self-E combines real-data local trajectory supervision with self-evaluation-based global distribution matching. Its hybrid objective supports inference with arbitrary step budgets.

3. Self-Evaluating Model

  • Self-E combines data-driven supervision with self-evaluation for global distribution matching, using the model's evolving estimates as feedback.The model is trained from scratch and does not require a separate pretrained teacher.Source: §3 · p. 3, §3.2 · p. 4

3.1. Learning from Data

  • When s=t, the data loss estimates E[x0|xt, c], while the expectation itself may not be a meaningful sample from q(x0|c).Training samples timestep pairs with s ≤ t.Source: §3.1 · p. 3

3.2. Learning by Self-Evaluation

  • When s<t, Self-E uses its data-trained model to approximate real-score feedback for global matching; classifier-score-only training suffices and improves convergence.This choice avoids co-training an additional fake-score model early in training.Source: §3.2 · p. 4

3.3. Final Objective

  • The final hybrid loss balances data reconstruction and self-evaluation; energy-preserving target normalization mitigates large-weight color bias and slightly improves visual quality and stability.When t=s, the objective reduces to a data-driven reconstruction loss.Source: §3.3 · pp. 4–5

3.4. Inference

  • At inference, Self-E iteratively removes noise using an arbitrary step budget; choosing s within [tk+1, tk] may improve results in some cases.The method uses energy-preserving classifier-free guidance with ω = 5.Source: §3.4 · p. 5

4. Experiment

Experiments compare Self-E with prior text-to-image models and use controlled ablations to assess alternative pretraining methods and design choices. Self-E leads across inference budgets, while its ablations identify a staged training strategy.

4. Experiment

  • The 2B, 512 × 512 comparison spans from-scratch, diffusion, flow-matching, and teacher-distilled models, while controlled ablations use 0.5B models under matched conditions.The ablations use 256 × 256 resolution and batch size 1024.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 6

4.1. Comparison with Prior Work

  • At 2 steps, FLUX, SANA, and SDXL fail to produce recognizable images, while LCM and TiM show structural and semantic degradation.At 50 steps, Self-E is competitive with FLUX despite FLUX being much larger.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2. Ablation Studies

  • Self-E consistently outperforms Flow Matching and IMM across step counts and maintains its advantage over Flow Matching throughout training.IMM is the native few-step baseline; Figure 4 presents qualitative comparisons and Figure 5 tracks GenEval scores over training.Source: §4.2 · p. 6

4.1. Comparison with Prior Work

  • GenEval Overall rises 0.753→0.781→0.785→0.815 from 2→4→8→50 steps, with a +0.12 2-step margin over the best prior methods.The paper reports Self-E as the top performer across all step counts.Source: §4.1 · p. 5, Table 1 · p. 7

4.2. Ablation Studies

  • Target normalization generally improves performance except at 2 steps, while adding the auxiliary term from the start degrades results.The final schedule uses classifier-score-only training early and adds the auxiliary term later, which helps mitigate two-step stripe artifacts.Source: §4.2 · pp. 6–7

5. Related Work

The related work frames diffusion and flow matching as strong generative approaches whose local prediction objectives typically require many inference steps. It also reviews solver and distillation strategies for faster sampling, alongside comparisons of training methods and design choices.

Controlled Ablation Study

  • The controlled ablation figure compares Self-E with Flow Matching and IMM across inference-step budgets and reports favorable results for Self-E.Source: Figure 4 · p. 8

Diffusion and Flow Matching

  • Diffusion and flow-matching models learn local score or velocity properties, and typically need many iterative denoising steps for high-quality samples.The paper notes their use across text-to-image, text-to-video, and language modeling.Source: §5 · pp. 7–8

Accelerating Diffusion/Flow Matching

  • Training-free high-order solvers approximate the underlying differential equations, but still struggle to produce high-quality samples within ten denoising steps.Source: §5 · p. 8

Training Progress Comparison

  • Training-progress plots compare GenEval scores at 2, 4, 8, and 50 steps from 50K to 300K iterations, with Self-E outperforming Flow Matching at each budget.Source: Figure 5 · p. 8

Auxiliary-Term Ablation

  • For the 2B model at two steps, classifier-score-only training shows checkerboard artifacts, while later auxiliary-term use helps mitigate them.Source: Figure 6 · p. 8

Accelerating Diffusion/Flow Matching

  • Distillation methods train students from pretrained teachers, including trajectory-transition, consistency, and flow-map approaches.Recent flow-map pretraining efforts such as TiM [59] and MeanFlow [15] remain difficult to scale effectively to text-to-image generation, according to the authors.Source: §5 · p. 8
  • Distribution-matching distillation moves student samples toward a teacher’s learned distribution using divergence-based losses at different noise levels.The paper positions its own approach as applying this distribution-matching perspective during text-to-image pretraining.Source: §5 · p. 8

6. Conclusion

Self-E is presented as a from-scratch, teacher-free text-to-image framework for flexible any-step inference. Its evaluations report strong few-step performance and monotonically improving results as inference steps increase.

6. Conclusion

  • Self-E uses dynamically learned local scores to assess generated samples and combine trajectory learning with global distribution matching.The conclusion reports state-of-the-art GenEval performance across inference budgets, particularly for few-step generation.Source: §6 · p. 9

Appendix

The appendix provides proofs, implementation details, additional experiments, prompt information, and a discussion of limitations and future work.

Appendix

  • Appendix sections cover loss-gradient proofs, training and inference implementation, second-timestep experiments, image prompts, and limitations with future directions.Source: Appendix · p. 12

S.1. Derivation of the Self-Evaluation Loss

The derivation starts from posterior means under the forward noising setup, then uses differences between those means to obtain relations for the self-evaluation loss.

S.1. Derivation of the Self-Evaluation Loss

  • The setup uses forward noising and takes gradients with respect to the re-noised prediction of x0.The model head matches the main text, and sg[·] denotes stop-gradient.Source: §S.1 · p. 12
  • Tweedie’s formula applied to the forward noising setup gives conditional and unconditional data posterior means and a conditional model posterior mean.The passage introduces these quantities as the starting point for the derivation.Source: §S.1 · p. 12
  • Subtracting the conditional and unconditional data posterior-mean expressions, then the conditional data and model expressions, yields two further relations.The derivation identifies these operations as subtracting the first two lines and the first and third lines, respectively.Source: §S.1 · p. 12

S.1.1. Self-evaluation without auxiliary term

Without the auxiliary term, Self-E uses a self-evaluation pseudo-target and squared loss; under posterior-mean approximations, its gradient follows the classifier score direction.

S.1.1. Self-evaluation without auxiliary term

  • The loss uses the self-evaluation pseudo-target from the main paper’s Eq. (14).The per-sample squared loss has expectation over (t, s, x0, ε) equal to Eq. (15).Source: §S.1.1 · p. 12
  • Under the posterior-mean approximation, gradient descent on the self-evaluation loss moves the re-noised prediction in the classifier-score direction.The direction is ∇_{x̂_s} log q(c|x̂_s).Source: §S.1.1 · p. 12
  • The derivation obtains the classifier-score update under approximations for the conditional and unconditional model posterior means.These approximate the model outputs by the corresponding expected x0 values.Source: §S.1.1 · p. 12

S.1.2. Self-evaluation with auxiliary term

With the auxiliary term, Self-E adds a fake-prompt branch and forms a self-evaluation target; the resulting gradient is proportional to the ideal vector field for a specified k.

S.1.2. Self-evaluation with auxiliary term

  • An optional fake-prompt branch estimates the model posterior mean for generated samples, and its output contributes to the self-evaluation target.The target is formed with a stop-gradient, then used in a per-sample squared loss.Source: §S.1.2 · p. 12
  • The auxiliary-term gradient is proportional to the full ideal vector field when k = (w − 1)/w; in practice, k is set to 0.9.The derivation uses the stated posterior-mean relations.Source: §S.1.2 · p. 12
  • The fake-prompt branch is trained by drawing model samples and reusing the conditional Flow Matching loss.The fake prompt concatenates “fake image” with the original prompt.Source: §S.1.2 · pp. 12–13
  • Training uses only the classifier term early, then enables the auxiliary term later to refine artifacts while retaining the stated overall objective.The schedule follows the main paper’s training schedule.Source: §S.1.2 · p. 13

S.2. Implementation Details

The implementation combines a FLUX-style latent transformer with a secondary time input and length-aware time scheduling. Training and inference details include a staged resolution schedule and DDIM-style sampling.

S.2. Implementation Details

  • A 2B-parameter model is trained first at approximately 256^2 resolution, then with higher-resolution data in a balanced 1:1 batch through 710k iterations.The initial stage runs for 500k iterations with batch size 1024; the later stage uses batch size 768 and introduces the auxiliary term at iteration 550k.Source: §S.2 · p. 13
  • The training setup uses Adam, a warmup and linear learning-rate decay, and an EMA model with decay 0.9999.The conditional branch uses EMA weights during the self-evaluation forward pass, while the unconditional branch uses non-EMA weights.Source: §S.2 · p. 13
  • The FLUX-style architecture uses a dual-time denoiser alongside a VAE, patchifier, and frozen text encoders.The 2B denoiser has 8 Double-Stream and 16 Single-Stream blocks; the ablation variant has 0.5B parameters.Source: §S.2 · p. 13
  • Frozen T5-XXL token embeddings and a pooled CLIP embedding provide text and global conditioning to the denoiser.Their outputs are linearly projected to the model dimension.Source: §S.2 · p. 13
  • The denoiser adds an auxiliary time input s for self-evaluation, encoded alongside t and the gap t − s.The combined embedding replaces the original single-time embedding, leaving downstream conditioning and modulation unchanged.Source: §S.2 · p. 13
  • The primary time is sampled from a logit-normal distribution, length-warped, while the secondary time equals t half the time and otherwise follows a training-dependent interval.The lower bound decreases from approximately t toward 0 during the first 300,000 iterations; the weighting function is 1/t^2.Source: §S.2 · p. 13
  • Supplementary figures show text-to-image samples at 2, 4, 8, and 50 inference steps, described as semantically coherent and strongly text-aligned.The showcased samples are described as diverse.Source: Figure S.1 · p. 14, Figure S.2 · p. 15
  • Inference uses a length-warped linear timestep scheduler and a DDIM-style update with η = 1.The stated settings identify η = 0 with deterministic DDIM and η = 1 with original DDPM ancestral sampling.Source: §S.2 · pp. 15–16

S.3. Additional Experimental Results

The supplement reports how secondary-timestep choices affect generation, provides additional inference-budget results and qualitative-result prompts, and discusses limitations and future directions.

S.3.2. More Results

  • Additional results at different inference budgets are provided in Figures S.1 and S.2.Source: §S.3.2 · p. 16

S.4. Prompts of Results

  • The supplement supplies prompts used for the main paper’s qualitative results.Source: §S.4 · p. 16

S.4.1. Prompts of Figure 1.

  • Figure 1 prompt examples are provided for 2-, 4-, 8-, and 50-step generation.The prompts describe varied scenes, including portraits, landscapes, and surreal images.Source: §S.4.1 · pp. 16–17

S.3.1. Alternative s-Scheduler

  • At s = 0.5, one-step generation without classifier-free guidance balances texture detail and image coherence, while both endpoint choices fail to produce meaningful images.The supplementary figure compares choices between s_k = t_k and s_k = t_k+1.Source: §S.3.1 · p. 16, §S.4.1 · p. 16, Figure S.4 · p. 17

S.4.2. Prompts of Figure 4.

  • Figure 4 prompt examples include chalkboard artwork, a home bar, a cat above mountains, and a royal guard.Source: §S.4.2 · p. 17

S.4.3. Prompts of Figure 5.

  • Figure 5 prompt examples describe a thunderhead over a wheat plain and a house made of luminous jelly bricks.Source: §S.4.3 · p. 17

S.5. Limitations and Future Work

  • At 1–2 inference steps, image details may lack sharpness compared with 50-step quality, and loss weighting and inference strategies remain under-optimized.The authors describe the paradigm as early in exploration.Source: §S.5 · p. 17
  • Future work includes improving training and inference scheduling, downstream fine-tuning, video adaptations, and unconditional generation.The authors characterize the training paradigm as distinct from consistency training and report coherent structure followed by iterative detail refinement.Source: §S.5 · pp. 17–18
Loading 2512.22374v1…