PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfLift: Accelerating Few-Step Diffusion via Self-Recovering Resolution Transition

Tingyan Wen, Chenqian Yan, Xurui Peng, Xiazhang Fang, Shuai Wang, Xueqian Wang, Songwei Liu

arXiv:2609.02036v1cs.CV

TL;DR

Few-step diffusion makes per-evaluation spatial cost increasingly important, while direct latent lifting can leave artifacts that limited remaining steps cannot reliably recover. SelfLift uses model-native consistency repair and on-policy self-recovery to support progressive-resolution inference, reducing latency by 41.5% and 44.1% and reaching 29.61× and 19.21× overall speedups with timestep distillation.

  • Problem

    Few-step diffusion leaves a limited recovery budget for artifacts and distribution mismatch introduced by direct latent lifting, constraining efficient late resolution transitions.

    Source: §1 · p. 2

  • Method

    SelfLift combines training-free Artifact-Aware Consistency Lift with On-Policy Self Recovery using model-native correction signals and an internal self-teacher on student-visited states.

    Source: §1 · p. 2

  • Results

    41.5% and 44.1% end-to-end latency reductions are achieved on FLUX.2-Klein and Z-Image-Turbo, with timestep distillation yielding 29.61× and 19.21× overall speedups over corresponding 50-step models.

    Source: §6 · p. 9

  • Takeaways & Limitations

    SelfLift provides a favorable speed-quality trade-off for aggressive few-step inference while avoiding external super-resolution and extra denoiser evaluations.

    Source: §1 · p. 2, §4.2 · p. 8

    SelfLift-rich depends on the quality of its internal self-teacher, making stronger self-teachers and trajectory-aligned supervision important directions for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Few-step diffusion models substantially compress temporal computation, making the spatial cost of each model evaluation an increasingly dominant source of inference latency. Progressive-resolution inference reduces this cost by performing early denoising at low resolution and reserving high-resolution computation for refinement. However, existing methods typically lift intermediate latents directly and rely on subsequent steps to absorb the induced distribution mismatch. In the few-step regime, the limited recovery budget leaves these errors as visible artifacts, constraining how late the transition can occur and, consequently, how efficiently it can be performed. We introduce SelfLift, a self-recovering progressive-resolution framework that derives both transition-repair signals and trajectory-aligned supervision from the generative model itself. SelfLift-zero proposes a training-free Artifact-Aware Consistency Lift, using disagreement between direct latent lifting and pixel-VAE re-encoding as both a localized artifact-risk signal and a model-native correction direction. It enables reliable late transitions without external super-resolution, extra denoiser evaluations, or sampling-schedule modifications. Building on this robust transition, SelfLift-rich performs On-Policy Self Recovery on student-visited states, transferring dense high-resolution guidance from an internal self-teacher while remaining aligned with the altered progressive-resolution dynamics. Across FLUX.2-Klein and Z-Image-Turbo, SelfLift reduces end-to-end latency by 41.5% and 44.1%, respectively. Combined with timestep distillation, it delivers overall speedups of 29.61x and 19.21x over the corresponding 50-step models while preserving competitive generation quality, establishing a stronger speed-quality frontier for few-step diffusion.

1 Introduction

Few-step diffusion shifts the acceleration bottleneck from temporal redundancy to per-step spatial cost, motivating progressive-resolution inference. SelfLift addresses transition artifacts and trajectory mismatch with model-native repair and supervision, achieving substantial latency reductions and speedups.

Motivation

  • Few-step distillation compresses sampling to one to eight evaluations, making per-step spatial cost a central acceleration bottleneck.Progressive-resolution inference addresses this by using low resolution early and reserving high-resolution computation for detail refinement.Source: §1 · pp. 1–2

Challenges

  • Direct latent lifting reduces overhead but can create visible artifacts and spatial drift that remaining high-resolution steps must recover.Later transitions accelerate more, yet leave a smaller recovery budget for correcting transition errors.Source: §1 · p. 2

SelfLift

  • SelfLift-zero introduces Artifact-Aware Consistency Lift, deriving artifact-risk localization and a correction direction from direct lifting versus pixel-VAE re-encoding.The training-free repair requires neither external super-resolution nor additional denoiser evaluations.Source: §1 · pp. 2–3
  • SelfLift-rich applies On-Policy Self Recovery to student-visited states, transferring dense high-resolution guidance from an internal self-teacher along the progressive-resolution trajectory.This targets the train–inference mismatch created when supervision uses states different from those visited by the accelerated rollout.Source: §1 · pp. 2–3

Results

  • 41.5% and 44.1% end-to-end latency reductions are reported on FLUX.2-Klein and Z-Image-Turbo, respectively.With timestep distillation, the corresponding overall speedups are 29.61× and 19.21× over 50-step models while maintaining strong generation quality.Source: §1 · p. 3

2 Related Work

Related work accelerates diffusion through temporal methods such as solvers, caching, and trajectory distillation, while spatial methods vary resolution along sampling. On-policy adaptation addresses the mismatch caused by supervising states that differ from those visited by distilled students.

Temporal acceleration

  • Temporal acceleration uses fast ODE solvers, error correction, multilevel methods, motion-aware caching, feature forecasting, and trajectory distillation.These approaches reduce temporal redundancy through faster integration, reuse, forecasting, or compressed sampling trajectories.Source: §2.1 · p. 3

Spatial acceleration

  • Spatial methods reduce inference cost by varying resolution along the sampling trajectory after temporal redundancy has been compressed.This positions progressive-resolution inference as a complementary direction to temporal acceleration.Source: §2.1 · p. 3

On-policy adaptation

  • Step-distilled models can be disrupted by standard post-training objectives, motivating adaptation that improves outputs without deviating from pretrained few-step dynamics.On-policy distillation supervises student-visited states to reduce train–inference mismatch from off-policy teacher data.Source: §2.2 · p. 3

3 Method

SelfLift performs progressive-resolution inference by lifting low-resolution states into a high-resolution trajectory, then repairing transition inconsistencies with model-derived anchors and selective correction. Its training-free and lightweight variants preserve the few-step dynamics while improving artifact suppression and detail recovery.

3.1 Preliminaries

  • Progressive-resolution inference follows the low-resolution flow, applies a cross-resolution lift at transition time, and completes the remaining trajectory at high resolution.The transition operator converts the low-resolution state into the high-resolution latent space.Source: §3.1 · p. 3

3.2 Analysis of Resolution Transition

  • Direct latent lifting can leave decoder-visible inconsistencies because its reachable subspace may differ from the native high-resolution VAE latent distribution.Directly lifted latents exhibit 18.93× native target-VAE round-trip energy, compared with 1.43× for pixel latents, and are associated with ghosting, structural distortion, and spatial drift.Source: §3.2 · p. 4
  • Pixel-VAE re-encoding provides an encoder-reachable but smoother anchor that retains coarse spatial information without uniquely recovering sample-specific high-frequency detail.When the conditional high-frequency distribution is non-degenerate, deterministic recovery has strictly positive irreducible error.Source: §3.2 · p. 4

3.3 SelfLift

  • SelfLift-zero computes the residual between direct lifting and pixel-VAE re-encoding, converts its magnitude into an artifact-risk map, and selectively corrects high-risk regions toward the anchor.Reliable locations remain on the direct-lift trajectory, while larger inconsistencies receive stronger correction, reducing artifacts without globally inheriting pixel-VAE smoothing.Source: §3.3 · pp. 4–5
  • SelfLift-zero re-noises the corrected clean estimate under the original flow marginal and resumes sampling at target resolution without changing the few-step schedule.The training-free method requires neither external super-resolution nor additional denoiser evaluations beyond the original schedule.Source: §3.3 · p. 5
  • SelfLift-rich distills the complete correction into a lightweight latent lifter and trains on student-visited states using internal high-resolution guidance and a dynamics objective.The rollout is advanced only by the student, while the dynamics loss preserves the pretrained few-step trajectory.Source: §3.3 · pp. 4–6, Figure 3 · p. 5

4 Experiments

SelfLift is evaluated on few-step and 50-step diffusion backbones through efficiency, quality, complementarity, and ablation studies. The results show that its training-free and learned variants improve the speed–quality trade-off while addressing transition artifacts and trajectory mismatch.

4.1 Settings

  • FLUX.2-Klein-9b uses 4 NFEs and Z-Image-Turbo uses 8 NFEs, with 512×512 low-resolution prefixes transitioning once to 1024×1024 outputs.The transition steps are tr = 3 and tr = 6, respectively.Source: §4.1 · p. 6

4.2 Main Results

  • 5.60×/4.58× compounded speedups with every quality drop below 5% versus Base (50) demonstrate complementarity between SelfLift and temporal acceleration.SelfLift-zero alone provides 1.92×/2.18× speedups on 50-step backbones, increasing when combined with TeaCache or TaylorSeer.Source: Table 2 · p. 7, §4.2 · p. 8
  • 41.5% and 44.1% latency reductions from SelfLift-rich yield 29.61× and 19.21× overall speedups with strong generation quality.SelfLift-rich consistently improves reported metrics over SelfLift-zero and attains the best ImageReward, PickScore, AestheticScore, and CLIPScore among accelerated methods on both backbones.Source: §4.2 · p. 8

4.3 Ablation Study

  • Later transitions improve acceleration but reduce recovery budget; SelfLift-zero preserves the strongest joint sharpness and prompt-aligned quality across both backbones.The consistency residual localizes artifact-prone regions for selective correction while preserving reliable latent details.Source: §4.3 · pp. 8–9, Figure 5 · p. 8, Figure 6 · p. 8
  • The full On-Policy Self Recovery model improves ImageReward from 1.1813 to 1.3171 and CLIPScore from 31.813 to 32.345 at the same inference cost.Off-policy supervision degrades quality, while on-policy supervision requires dynamics preservation for the full improvement.Source: Table 3 · p. 8, §4.3 · p. 9

5 Limitations and Discussion

SelfLift relies on capabilities increasingly available in modern multimodal generative models, and SelfLift-rich depends on the quality of its internal self-teacher. Strengthening self-teachers and trajectory-aligned supervision remains future work.

5 Limitations and Discussion

  • SelfLift’s self-recovering design assumes native multi-resolution support and image-conditioned editing in compatible multimodal generative models.SelfLift-zero remains training-free and plug-and-play across compatible model families.Source: §5 · p. 9
  • SelfLift-rich effectiveness depends on the quality of its internal self-teacher, despite prompt–reference curation used to improve its guidance.The design avoids a separately trained teacher or external reward model, but stronger self-teachers remain an open direction.Source: §5 · p. 9

6 Conclusion

SelfLift combines a training-free artifact-aware transition with learned on-policy recovery for progressive-resolution few-step inference, reducing latency while completing high-resolution refinement.

  • SelfLift-zero provides a training-free Artifact-Aware Consistency Lift for reliable resolution transitions without external super-resolution or extra denoising evaluations.Source: §6 · p. 9
  • SelfLift inference follows a low-resolution trajectory to the transition timestep, constructs a target-resolution state, re-noises it, and completes high-resolution sampling.Source: A SelfLift Algorithm · p. 12
  • SelfLift-rich distills the zero transition into a latent lifter and applies On-Policy Self Recovery on student-visited states with dense high-resolution supervision.Source: A SelfLift Algorithm · p. 12
  • SelfLift-zero forms paired lifted and pixel-VAE states, computes their consistency residual, and uses adaptive artifact-aware correction; SelfLift-rich instead applies the distilled lifter.Source: A SelfLift Algorithm · p. 12
  • SelfLift-rich training uses a latent lifter-distillation stage followed by on-policy recovery training with a student LoRA and dynamics-preserving objective.Source: A SelfLift Algorithm · p. 12

B Consistency Analysis

The consistency analysis shows that direct latent lifting produces VAE-inconsistent representations, while pixel-VAE re-encoding remains close to stable native high-resolution latents.

  • Direct latent lifting preserves trajectory continuity but can create decoder-visible inconsistencies because it mismatches the native high-resolution VAE latent distribution.Source: B Consistency Analysis · p. 12
  • A stable VAE-consistent latent changes little after target-VAE decode–encode round trips, whereas an unsupported latent undergoes an abnormally large change.Source: B Consistency Analysis · p. 13
  • The calibration compares native high-resolution, directly lifted, and pixel-VAE latents across 1,000 native 1024 × 1024 images using a 512 × 512 low-resolution stage.Source: B Consistency Analysis · p. 13
  • 0.35859: direct latent lifting has an abnormally large round-trip energy, 18.93× the native baseline and 13.20× the pixel-VAE energy.Source: B Consistency Analysis · p. 13
  • Pixel-VAE re-encoding closes 97.4% of the direct-lift–native energy gap on average, while direct lifting shifts latents toward unstable regions that can manifest as ghosting and spatial drift.Source: B Consistency Analysis · p. 13, Figure 7 · p. 13

C.2 Baseline Implementation Details

The appendix specifies evaluation protocols, baseline configurations, latency measurement, and SelfLift-zero parameter studies for reproducible few-step comparisons.

  • Spatial baselines generally use a 512 × 512 low-resolution stage before transitioning to 1024 × 1024 at tr = 3 or tr = 6, while MrFlow retains pixel-space super-resolution.Source: §C.2 · pp. 13–14
  • Inference efficiency uses one GPU, 10 warm-up runs, and the average of 50 generations; speedup is relative to Base (50), while latency reduction is relative to Base (4)/(8).Source: §C.3 · p. 14, §C.4 · p. 15
  • The evaluation measures ImageReward, PickScore, AestheticScore, CLIPScore, GenEval, and CLIP-IQA for overall quality, preference, aesthetics, alignment, instruction following, and transition sharpness.Source: §C.3 · p. 14
  • 41.5%: SelfLift-rich reduces end-to-end latency relative to Base (4), while its 3-LR+1-HR schedule reduces denoising latency by 56.0%.Source: Figure 8 · p. 15, Table 7 · p. 15
  • Later transitions increase low-resolution computation savings but leave fewer high-resolution steps for recovering transition errors.Source: §D.1 · p. 15, §D.1.2 · p. 16, Table 8 · p. 16
  • Increasing the correction region improves stability but can smooth outputs; the selected settings are ρ = 0.4 for FLUX.2-Klein-9b and ρ = 0.3 for Z-Image-Turbo.Source: §D.1.1 · p. 15, §D.1 · p. 15, Table 10 · p. 15

D.2 Ablation of On-Policy Self Recovery

The ablation isolates learned lifting, on-policy supervision, and dynamics preservation, showing that their combination supports sharp recovery while avoiding transition artifacts and latency overhead.

  • The ablation separates a learned lifter, trajectory-aligned on-policy supervision, and dynamics preservation, with all learned variants using tr = 3.Source: §D.2 · pp. 16–17
  • The learned lifter removes text ghosting and matches SelfLift-zero without requiring pixel–VAE inference.Source: Figure 9 · p. 16, §D.2 · p. 17
  • Off-policy supervision causes severe blur, while removing dynamics preservation yields incomplete recovery; the full model remains sharp.Source: Figure 9 · p. 16, §D.2 · p. 17
  • 1.556 seconds: the learned lifter reduces transition latency from 1.642 to 1.556 seconds without degrading quality.Source: §D.2 · p. 17
  • λdyn = 80 performs best; λdyn = 40 under-regularizes dynamics, whereas λdyn = 120 limits adaptation and approaches SelfLift-zero.Source: §D.2 · p. 17
  • Localized correction removes artifacts while preserving reliable content, whereas uniform mixing leaves residual distortions.Source: §D.3 · p. 17, Figure 10 · p. 17

D.4 Additional Qualitative Results

Additional prompts across FLUX.2-Klein-9b and Z-Image-Turbo show that SelfLift consistently suppresses transition artifacts while preserving text accuracy, local structure, and fine-grained visual details.

  • SelfLift consistently suppresses transition artifacts across additional prompts and model families.Source: §D.4 · p. 17
  • SelfLift preserves text accuracy, local structure, and fine-grained visual details.Source: §D.4 · p. 17

E Applicability and practical adaptation.

SelfLift-zero is most suited to aggressive late transitions, while video use should remain within the backbone’s native multi-resolution regime.

Related papers

Loading 2609.02036v1…