Source-linked AI summary
SelfLift: Accelerating Few-Step Diffusion via Self-Recovering Resolution Transition
Tingyan Wen, Chenqian Yan, Xurui Peng, Xiazhang Fang, Shuai Wang, Xueqian Wang, Songwei Liu
TL;DR
Few-step diffusion makes per-evaluation spatial cost increasingly important, while direct latent lifting can leave artifacts that limited remaining steps cannot reliably recover. SelfLift uses model-native consistency repair and on-policy self-recovery to support progressive-resolution inference, reducing latency by 41.5% and 44.1% and reaching 29.61× and 19.21× overall speedups with timestep distillation.
Problem
Few-step diffusion leaves a limited recovery budget for artifacts and distribution mismatch introduced by direct latent lifting, constraining efficient late resolution transitions.
Method
SelfLift combines training-free Artifact-Aware Consistency Lift with On-Policy Self Recovery using model-native correction signals and an internal self-teacher on student-visited states.
Results
41.5% and 44.1% end-to-end latency reductions are achieved on FLUX.2-Klein and Z-Image-Turbo, with timestep distillation yielding 29.61× and 19.21× overall speedups over corresponding 50-step models.
Takeaways & Limitations
SelfLift provides a favorable speed-quality trade-off for aggressive few-step inference while avoiding external super-resolution and extra denoiser evaluations.
Takeaways & Limitations
SelfLift-rich depends on the quality of its internal self-teacher, making stronger self-teachers and trajectory-aligned supervision important directions for future work.
Abstract
from arXiv · showhide
Few-step diffusion models substantially compress temporal computation, making the spatial cost of each model evaluation an increasingly dominant source of inference latency. Progressive-resolution inference reduces this cost by performing early denoising at low resolution and reserving high-resolution computation for refinement. However, existing methods typically lift intermediate latents directly and rely on subsequent steps to absorb the induced distribution mismatch. In the few-step regime, the limited recovery budget leaves these errors as visible artifacts, constraining how late the transition can occur and, consequently, how efficiently it can be performed. We introduce SelfLift, a self-recovering progressive-resolution framework that derives both transition-repair signals and trajectory-aligned supervision from the generative model itself. SelfLift-zero proposes a training-free Artifact-Aware Consistency Lift, using disagreement between direct latent lifting and pixel-VAE re-encoding as both a localized artifact-risk signal and a model-native correction direction. It enables reliable late transitions without external super-resolution, extra denoiser evaluations, or sampling-schedule modifications. Building on this robust transition, SelfLift-rich performs On-Policy Self Recovery on student-visited states, transferring dense high-resolution guidance from an internal self-teacher while remaining aligned with the altered progressive-resolution dynamics. Across FLUX.2-Klein and Z-Image-Turbo, SelfLift reduces end-to-end latency by 41.5% and 44.1%, respectively. Combined with timestep distillation, it delivers overall speedups of 29.61x and 19.21x over the corresponding 50-step models while preserving competitive generation quality, establishing a stronger speed-quality frontier for few-step diffusion.
1 Introduction
Few-step diffusion shifts the acceleration bottleneck from temporal redundancy to per-step spatial cost, motivating progressive-resolution inference. SelfLift addresses transition artifacts and trajectory mismatch with model-native repair and supervision, achieving substantial latency reductions and speedups.
- Motivation: Few-step distillation compresses sampling to one to eight evaluations, making per-step spatial cost a central acceleration bottleneck.Progressive-resolution inference addresses this by using low resolution early and reserving high-resolution computation for detail refinement.
- Challenges: Direct latent lifting reduces overhead but can create visible artifacts and spatial drift that remaining high-resolution steps must recover.Later transitions accelerate more, yet leave a smaller recovery budget for correcting transition errors.
- SelfLift: SelfLift-zero introduces Artifact-Aware Consistency Lift, deriving artifact-risk localization and a correction direction from direct lifting versus pixel-VAE re-encoding.The training-free repair requires neither external super-resolution nor additional denoiser evaluations.
- SelfLift: SelfLift-rich applies On-Policy Self Recovery to student-visited states, transferring dense high-resolution guidance from an internal self-teacher along the progressive-resolution trajectory.This targets the train–inference mismatch created when supervision uses states different from those visited by the accelerated rollout.
- Results: 41.5% and 44.1% end-to-end latency reductions are reported on FLUX.2-Klein and Z-Image-Turbo, respectively.With timestep distillation, the corresponding overall speedups are 29.61× and 19.21× over 50-step models while maintaining strong generation quality.
2 Related Work
Related work accelerates diffusion through temporal methods such as solvers, caching, and trajectory distillation, while spatial methods vary resolution along sampling. On-policy adaptation addresses the mismatch caused by supervising states that differ from those visited by distilled students.
- Temporal acceleration: Temporal acceleration uses fast ODE solvers, error correction, multilevel methods, motion-aware caching, feature forecasting, and trajectory distillation.These approaches reduce temporal redundancy through faster integration, reuse, forecasting, or compressed sampling trajectories.
- Spatial acceleration: Spatial methods reduce inference cost by varying resolution along the sampling trajectory after temporal redundancy has been compressed.This positions progressive-resolution inference as a complementary direction to temporal acceleration.
- On-policy adaptation: Step-distilled models can be disrupted by standard post-training objectives, motivating adaptation that improves outputs without deviating from pretrained few-step dynamics.On-policy distillation supervises student-visited states to reduce train–inference mismatch from off-policy teacher data.
3 Method
SelfLift performs progressive-resolution inference by lifting low-resolution states into a high-resolution trajectory, then repairing transition inconsistencies with model-derived anchors and selective correction. Its training-free and lightweight variants preserve the few-step dynamics while improving artifact suppression and detail recovery.
- 3.1 Preliminaries: Progressive-resolution inference follows the low-resolution flow, applies a cross-resolution lift at transition time, and completes the remaining trajectory at high resolution.The transition operator converts the low-resolution state into the high-resolution latent space.
- 3.2 Analysis of Resolution Transition: Direct latent lifting can leave decoder-visible inconsistencies because its reachable subspace may differ from the native high-resolution VAE latent distribution.Directly lifted latents exhibit 18.93× native target-VAE round-trip energy, compared with 1.43× for pixel latents, and are associated with ghosting, structural distortion, and spatial drift.
- 3.2 Analysis of Resolution Transition: Pixel-VAE re-encoding provides an encoder-reachable but smoother anchor that retains coarse spatial information without uniquely recovering sample-specific high-frequency detail.When the conditional high-frequency distribution is non-degenerate, deterministic recovery has strictly positive irreducible error.
- 3.3 SelfLift: SelfLift-zero computes the residual between direct lifting and pixel-VAE re-encoding, converts its magnitude into an artifact-risk map, and selectively corrects high-risk regions toward the anchor.Reliable locations remain on the direct-lift trajectory, while larger inconsistencies receive stronger correction, reducing artifacts without globally inheriting pixel-VAE smoothing.
- 3.3 SelfLift: SelfLift-zero re-noises the corrected clean estimate under the original flow marginal and resumes sampling at target resolution without changing the few-step schedule.The training-free method requires neither external super-resolution nor additional denoiser evaluations beyond the original schedule.
- 3.3 SelfLift: SelfLift-rich distills the complete correction into a lightweight latent lifter and trains on student-visited states using internal high-resolution guidance and a dynamics objective.The rollout is advanced only by the student, while the dynamics loss preserves the pretrained few-step trajectory.
4 Experiments
SelfLift is evaluated on few-step and 50-step diffusion backbones through efficiency, quality, complementarity, and ablation studies. The results show that its training-free and learned variants improve the speed–quality trade-off while addressing transition artifacts and trajectory mismatch.
- 4.1 Settings: FLUX.2-Klein-9b uses 4 NFEs and Z-Image-Turbo uses 8 NFEs, with 512×512 low-resolution prefixes transitioning once to 1024×1024 outputs.The transition steps are tr = 3 and tr = 6, respectively.
- 4.2 Main Results: 5.60×/4.58× compounded speedups with every quality drop below 5% versus Base (50) demonstrate complementarity between SelfLift and temporal acceleration.SelfLift-zero alone provides 1.92×/2.18× speedups on 50-step backbones, increasing when combined with TeaCache or TaylorSeer.
- 4.2 Main Results: 41.5% and 44.1% latency reductions from SelfLift-rich yield 29.61× and 19.21× overall speedups with strong generation quality.SelfLift-rich consistently improves reported metrics over SelfLift-zero and attains the best ImageReward, PickScore, AestheticScore, and CLIPScore among accelerated methods on both backbones.
- 4.3 Ablation Study: Later transitions improve acceleration but reduce recovery budget; SelfLift-zero preserves the strongest joint sharpness and prompt-aligned quality across both backbones.The consistency residual localizes artifact-prone regions for selective correction while preserving reliable latent details.
- 4.3 Ablation Study: The full On-Policy Self Recovery model improves ImageReward from 1.1813 to 1.3171 and CLIPScore from 31.813 to 32.345 at the same inference cost.Off-policy supervision degrades quality, while on-policy supervision requires dynamics preservation for the full improvement.
5 Limitations and Discussion
SelfLift relies on capabilities increasingly available in modern multimodal generative models, and SelfLift-rich depends on the quality of its internal self-teacher. Strengthening self-teachers and trajectory-aligned supervision remains future work.
- 5 Limitations and Discussion: SelfLift’s self-recovering design assumes native multi-resolution support and image-conditioned editing in compatible multimodal generative models.SelfLift-zero remains training-free and plug-and-play across compatible model families.
- 5 Limitations and Discussion: SelfLift-rich effectiveness depends on the quality of its internal self-teacher, despite prompt–reference curation used to improve its guidance.The design avoids a separately trained teacher or external reward model, but stronger self-teachers remain an open direction.
6 Conclusion
SelfLift combines a training-free artifact-aware transition with learned on-policy recovery for progressive-resolution few-step inference, reducing latency while completing high-resolution refinement.
- SelfLift-zero provides a training-free Artifact-Aware Consistency Lift for reliable resolution transitions without external super-resolution or extra denoising evaluations.
- SelfLift inference follows a low-resolution trajectory to the transition timestep, constructs a target-resolution state, re-noises it, and completes high-resolution sampling.
- SelfLift-rich distills the zero transition into a latent lifter and applies On-Policy Self Recovery on student-visited states with dense high-resolution supervision.
- SelfLift-zero forms paired lifted and pixel-VAE states, computes their consistency residual, and uses adaptive artifact-aware correction; SelfLift-rich instead applies the distilled lifter.
- SelfLift-rich training uses a latent lifter-distillation stage followed by on-policy recovery training with a student LoRA and dynamics-preserving objective.
B Consistency Analysis
The consistency analysis shows that direct latent lifting produces VAE-inconsistent representations, while pixel-VAE re-encoding remains close to stable native high-resolution latents.
- Direct latent lifting preserves trajectory continuity but can create decoder-visible inconsistencies because it mismatches the native high-resolution VAE latent distribution.
- A stable VAE-consistent latent changes little after target-VAE decode–encode round trips, whereas an unsupported latent undergoes an abnormally large change.
- The calibration compares native high-resolution, directly lifted, and pixel-VAE latents across 1,000 native 1024 × 1024 images using a 512 × 512 low-resolution stage.
- 0.35859: direct latent lifting has an abnormally large round-trip energy, 18.93× the native baseline and 13.20× the pixel-VAE energy.
- Pixel-VAE re-encoding closes 97.4% of the direct-lift–native energy gap on average, while direct lifting shifts latents toward unstable regions that can manifest as ghosting and spatial drift.
C.2 Baseline Implementation Details
The appendix specifies evaluation protocols, baseline configurations, latency measurement, and SelfLift-zero parameter studies for reproducible few-step comparisons.
- Spatial baselines generally use a 512 × 512 low-resolution stage before transitioning to 1024 × 1024 at tr = 3 or tr = 6, while MrFlow retains pixel-space super-resolution.
- Inference efficiency uses one GPU, 10 warm-up runs, and the average of 50 generations; speedup is relative to Base (50), while latency reduction is relative to Base (4)/(8).
- The evaluation measures ImageReward, PickScore, AestheticScore, CLIPScore, GenEval, and CLIP-IQA for overall quality, preference, aesthetics, alignment, instruction following, and transition sharpness.
- 41.5%: SelfLift-rich reduces end-to-end latency relative to Base (4), while its 3-LR+1-HR schedule reduces denoising latency by 56.0%.
- Later transitions increase low-resolution computation savings but leave fewer high-resolution steps for recovering transition errors.
- Increasing the correction region improves stability but can smooth outputs; the selected settings are ρ = 0.4 for FLUX.2-Klein-9b and ρ = 0.3 for Z-Image-Turbo.
D.2 Ablation of On-Policy Self Recovery
The ablation isolates learned lifting, on-policy supervision, and dynamics preservation, showing that their combination supports sharp recovery while avoiding transition artifacts and latency overhead.
- The ablation separates a learned lifter, trajectory-aligned on-policy supervision, and dynamics preservation, with all learned variants using tr = 3.
- The learned lifter removes text ghosting and matches SelfLift-zero without requiring pixel–VAE inference.
- Off-policy supervision causes severe blur, while removing dynamics preservation yields incomplete recovery; the full model remains sharp.
- 1.556 seconds: the learned lifter reduces transition latency from 1.642 to 1.556 seconds without degrading quality.
- λdyn = 80 performs best; λdyn = 40 under-regularizes dynamics, whereas λdyn = 120 limits adaptation and approaches SelfLift-zero.
- Localized correction removes artifacts while preserving reliable content, whereas uniform mixing leaves residual distortions.
D.4 Additional Qualitative Results
Additional prompts across FLUX.2-Klein-9b and Z-Image-Turbo show that SelfLift consistently suppresses transition artifacts while preserving text accuracy, local structure, and fine-grained visual details.
- SelfLift consistently suppresses transition artifacts across additional prompts and model families.
- SelfLift preserves text accuracy, local structure, and fine-grained visual details.
E Applicability and practical adaptation.
SelfLift-zero is most suited to aggressive late transitions, while video use should remain within the backbone’s native multi-resolution regime.
- SelfLift-zero is better suited to aggressive late transitions because clean-sample estimates and pixel-VAE guidance become more reliable later.
- Transitioning too early can preserve residual noise or incomplete structure, producing an under-denoised appearance and reducing acceleration.
- Video application should remain within the model’s native multi-resolution regime because unseen sequence lengths can destabilize scene structure.