PaperlayerView exact revision on arXiv

Source-linked AI summary

One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models

Aleksandr Razin, Danil Kazantsev, Ilya Makarov

arXiv:2511.10629v2cs.CV

TL;DR

Diffusion generators are constrained by their training resolutions, while latent resizing and image-space super-resolution bring artifacts or latency. LUA upsamples the latent before a single final decode; at 2048^2 and 4096^2, SDXL+LUA achieves state-of-the-art single-decode fidelity while remaining fastest.

  • Problem

    High-resolution synthesis is limited by distortions beyond training scales, and preserving latent detail during upscaling without an extra diffusion process remains a challenge.

    Source: §1 · pp. 1–2

  • Method

    LUA is a trained latent upscaler with a shared backbone and scale-specific ×2/×4 heads, designed to work across VAEs without retraining the generator.

    Source: §1 · p. 2

  • Results

    At 2048^2 and 4096^2, SDXL+LUA achieves state-of-the-art single-decode fidelity (FID 180.80 / 176.90; pFID 97.90 / 61.80) while remaining fastest.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The adapter transfers across SDXL, SD3, and FLUX with minimal fine-tuning, without modifying the generator or adding diffusion stages.

    Source: §1 · p. 2, §4.3 · p. 7

    Errors or biases in the generator’s latent are faithfully upscaled, so artifacts in the base sample may persist at higher resolution.

    Source: §5 · p. 7

Abstract

from arXiv · show

Diffusion models struggle to scale beyond their training resolutions, as direct high-resolution sampling is slow and costly, while post-hoc image super-resolution (ISR) introduces artifacts and additional latency by operating after decoding. We present the Latent Upscaler Adapter (LUA), a lightweight module that performs super-resolution directly on the generator's latent code before the final VAE decoding step. LUA integrates as a drop-in component, requiring no modifications to the base model or additional diffusion stages, and enables high-resolution synthesis through a single feed-forward pass in latent space. A shared Swin-style backbone with scale-specific pixel-shuffle heads supports 2x and 4x factors and remains compatible with image-space SR baselines, achieving comparable perceptual quality with nearly 3x lower decoding and upscaling time (adding only +0.42 s for 1024 px generation from 512 px, compared to 1.87 s for pixel-space SR using the same SwinIR architecture). Furthermore, LUA shows strong generalization across the latent spaces of different VAEs, making it easy to deploy without retraining from scratch for each new decoder. Extensive experiments demonstrate that LUA closely matches the fidelity of native high-resolution generation while offering a practical and efficient path to scalable, high-fidelity image synthesis in modern diffusion pipelines.

1. Introduction

The introduction frames high-resolution diffusion as constrained by training scales and costly or artifact-prone upscaling. It presents LUA as a single-pass latent adapter and evaluates its quality, efficiency, multi-scale support, and transfer across models.

Introduction

  • Training beyond typical 512^2 or 1024^2 scales can cause repetition, geometric distortions, and texture breakdown, while retraining or high-resolution fine-tuning demands substantial compute and data.Source: §1 · p. 1
  • Reference-based latent upscaling methods such as DemoFusion-style pipelines and LSRNA require a second diffusion stage, auxiliary guidance, and tight coupling to specific VAEs.Source: §1 · pp. 1–2
  • Naïve latent resizing departs from the valid-latent manifold, while decoded-image SR can improve fidelity but adds latency and may introduce noise.Source: §1 · pp. 1–2
  • The study tests whether a simple adapter can improve quality and cost against native high-resolution synthesis and image-space SR, support multiple scales, and transfer across VAEs.It evaluates across backbones and resolutions using FID, KID, CLIP, and local-detail metrics, with architecture, objective, and scale-handling ablations.Source: §1 · p. 2
  • LUA uses a shared backbone and scale-specific heads to upscale latents by ×2 or ×4 before one final decode, without retraining the generator or adding a diffusion stage.The paper reports that it is more efficient and less noisy than pixel-space SR while achieving quality comparable to modern high-resolution diffusion pipelines.Source: §1 · p. 2

2. Related Work

Prior approaches to high-resolution synthesis trade off compute, fidelity, and flexibility: some use additional diffusion passes, while pixel-space SR and latent resizing have distinct costs and artifacts. LUA targets a single-pass latent alternative with shared multi-scale capacity.

High-resolution diffusion

  • Inference-time high-resolution diffusion strategies avoid full retraining, but tiling can risk seams, extreme-resolution speed methods can reduce fidelity, and progressive or reference-based pipelines add diffusion passes and latency.Source: §2 · pp. 2–3

Super-resolution in image and latent spaces

  • Pixel-space SR methods denoise at target resolution, incurring quadratic compute and memory with image size and risking semantic drift, while learned latent mappings may still require a subsequent diffusion stage.Naïve latent interpolation can also depart from the generative manifold.Source: §2 · p. 3

Discrete vs. continuous multi-scale SR

  • Discrete-factor SR often uses separate models or shared backbones with scale-specific heads, whereas arbitrary-scale coordinate methods can underperform on fine textures.Source: §2 · p. 3

LUA design motivation

  • LUA targets single-pass latent upscaling with a dedicated training curriculum and shared backbone with lightweight scale-specific heads, avoiding extra diffusion stages and pixel-space SR’s quadratic cost and drift.Source: §2 · p. 3

3. Proposed Method

LUA enlarges a generator’s latent before decoding, using a shared multi-scale architecture and cross-VAE adaptation. Its multi-stage latent–pixel curriculum links latent structure to decoded appearance while keeping synthesis single-decode.

3. Proposed Method

  • The proposed adapter sits between a pretrained generator and frozen decoder, enlarges the latent, and uses one decode; cross-model transfer changes only the first convolution and needs minimal fine-tuning.Source: §3 · p. 3, Figure 3 · p. 3

3.1. Latent Upscaling

  • The deterministic upscaler maps a latent to a ×2 or ×4 latent while remaining on the latent manifold; with decoder stride s=8, it processes about 1/64 as many spatial elements as image-space SR.It still decodes once and adds no second diffusion pass or full-resolution refinement stage.Source: §3.1 · pp. 3–4

3.2. Architecture

  • A Swin Transformer restoration backbone extracts latent features and feeds scale-specific pixel-shuffle heads; joint training shares the backbone across ×2 and ×4 factors, with one selected head at inference.For VAEs with different latent channel widths, only the first convolution changes; the shared backbone and heads are reused after brief fine-tuning.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. Multi-stage Training Strategy

  • The three-stage curriculum first aligns latent structure and spectra, then adds decoded-image consistency and high-frequency supervision, and finally refines edges in pixel space without re-diffusion.This balances latent and pixel objectives because latent-only training leaves decoded artifacts, while pixel-only optimization can fail to converge; edge-aware loss helps reduce staircase artifacts.Source: §3.3 · pp. 4–5

4. Experiments

LUA is evaluated across resolutions, models, and scales against high-resolution and super-resolution baselines, with protocol and ablations examining its efficiency, fidelity, and design. Results show its strongest fidelity at 2K/4K, transfer across model families, and qualitative texture advantages, while ablations favor the full curriculum and shared multi-head design.

4.1. Experiment Settings

  • Training uses 3.8 million OpenImages crop pairs and three 125,000-step curriculum stages; cross-VAE transfer reuses the backbone and heads after replacing the first convolution and brief fine-tuning.The shared ×2/×4 backbone is trained jointly with balanced sampling.Source: §4.1 · p. 5

4.2. Evaluation Protocol

  • Evaluation uses 1,000 held-out OpenImages photos, reports image and patch fidelity metrics plus CLIP and runtime, and compares SDXL+LUA with direct, pixel-SR, and diffusion-based baselines.Runtime is median per-image wall-clock time after 20 warm-ups on one GPU; composite timing includes generation, upscaling, and decode.Source: §4.2 · p. 6

4.3. Quantitative Results

  • At 2048², SDXL+LUA records FID 180.80, pFID 97.90, KID 0.0018, and CLIP 0.764 in 3.52 s; at 4096², it records FID 176.90 and pFID 61.80 in 6.87 s.At both resolutions, the cited results report stronger single-decode fidelity and faster runtime than SDXL+SwinIR; at 4096², direct SDXL sampling has FID 280.42.Source: §4.3 · p. 6, Table 1 · p. 6
  • Across FLUX, SDXL, and SD3, LUA transfers across ×2 and ×4 scales with reported gains and no generator retraining or added diffusion stages.At ×2, FLUX+LUA reaches FID 180.99 and CLIP 0.773; at ×4, SDXL+LUA reaches KID 0.0015 and FLUX+LUA has the strongest pFID, 62.30.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4. Qualitative Results

  • LUA preserves edge continuity and microstructure with fewer artifacts than pixel-space SwinIR, while avoiding the second diffusion stage used by DemoFusion+LSRNA.Examples include distinct fur strands and coherent petal detail without SwinIR’s halos or granular noise.Source: §4.4 · p. 7

4.5. Ablation Studies

  • The full three-stage curriculum gives the best reported fidelity at both ×2 and ×4, while removing stages weakens perceptual quality or latent–pixel consistency.At ×2, the full model scores PSNR 32.54 / LPIPS 0.138 versus 28.96 / 0.172 for I+II without Stage III; a pixel-only variant fails to converge with a frozen VAE.Source: §4.5 · p. 7, Table 3 · p. 7
  • The joint multi-head design achieves the best PSNR/LPIPS at ×2 and ×4 while consolidating capacity in one shared backbone.The alternatives are LIIF and separate single-scale specialist models.Source: §4.5 · p. 7, Table 4 · p. 7

5. Discussion

LUA faithfully upscales errors or biases already present in the generator’s latent, so some base-sample artifacts can persist at higher resolution. The discussion proposes latent refinement, image-to-image applications, and temporal consistency for future video use.

5. Discussion

  • Errors or biases in the generator latent are faithfully upscaled, so artifacts in the base sample can persist at higher resolution.The discussion proposes latent-space refinement, potentially gated by uncertainty, as a future direction.Source: §5 · pp. 7–8
  • The adapter’s mechanism may also serve image-to-image tasks such as depth-to-image or semantic maps, where preserving structure during upscaling is critical.The discussion identifies these as possible extensions beyond text-to-image.Source: §5 · p. 8
  • Video extension remains a future need: practical high-resolution synthesis in dynamic settings requires temporal consistency, such as recurrent latent refinement or temporal priors.The paper names these as possible approaches.Source: §5 · p. 8

6. Conclusion

LUA performs single-pass latent upscaling before decoding and achieves strong single-decode fidelity at 2048² and 4096² while remaining fast. Its performance is constrained at 1024² by the input-latent resolution, and the adapter transfers across several VAE families with brief fine-tuning.

6. Conclusion

  • At 2048² and 4096², SDXL+LUA reports FID 180.80 / 176.90 and pFID 97.90 / 61.80, achieving state-of-the-art single-decode fidelity while remaining the fastest.At 2048², it runs in 3.52 s versus 7.23 s for SDXL (Direct), outperforms pixel-space SR, and approaches multi-stage re-diffusion quality at a fraction of the runtime.Source: §6 · p. 8
  • At 1024², performance trails the strongest baselines because the input latent is constrained to 64×64, though patch-level fidelity remains competitive.Source: §6 · p. 8
  • A single LUA backbone transfers across SDXL, SD3, and FLUX with only a first-layer channel change and brief fine-tuning.The adapter uses a multi-stage latent–pixel curriculum and scale-specific ×2/×4 heads, without modifying the generator or adding diffusion stages.Source: §6 · p. 8
  • The results establish single-decode latent upscaling as a practical alternative to multi-stage high-resolution pipelines.Source: §6 · p. 8

Supplementary Material

The supplement details a three-stage training curriculum, its loss terms, and optimizer settings. It also illustrates the losses and artifacts associated with training without latent normalization.

A.1. Per-stage Loss Weights

  • Stage I aligns latents with reconstruction and spectral losses; Stage II adds pixel-domain consistency and high-frequency terms; Stage III refines decoded pixels with reconstruction, frequency, and EAGLE losses.The loss visualizations show prediction, ground truth, error maps, and loss formulas; brighter regions indicate larger gradient contributions.Source: §A.1 · p. 11
  • Stage II uses α2 = 1.0, β2 = 0.1, γ2 = 0.1, and δ2 = 0.05; its added terms include bicubic downsample consistency and Gaussian high-frequency residual loss.The downsample-consistency term compares images after both are downsampled by a factor of 2; the high-frequency loss uses a Gaussian kernel of size 5 and σ = 1.0.Source: §A.1 · p. 11
  • EAGLE is used as a light regularizer to sharpen edges and suppress grid artifacts during pixel refinement.It compares frequency-filtered patch-variance maps of horizontal and vertical image gradients; the Gaussian high-pass mask uses normalized spatial frequencies and cutoff fc = 0.5.Source: §A.1 · p. 11
  • The supplement also provides a visual overview of the staged losses and their error maps.Source: Figure 7 · p. 12

A.2. Optimization and Scheduler Settings

  • Each stage uses 125,000 training steps with Adam at learning rate 2 × 10^-4, a MultiStepLR schedule, and EMA weights for validation and final reporting.The scheduler milestones are 62,500, 93,750, and 112,500 steps, with decay factor γ = 0.5 and no warm-up.Source: §A.2 · p. 12

A.3. Latent Normalization and Limitations

With a frozen VAE decoder, direct pixel-only training from scratch can destabilize LUA; the three-stage curriculum mitigates this by aligning latent statistics before pixel-domain optimization. The adapter still lacks an explicit bound on its output latents, leaving a robustness limitation under distribution shift.

A.3. Latent Normalization and Limitations

  • With a frozen VAE decoder, pixel-only training from scratch can drive latents outside the decoder’s typical range, causing saturated codes, broken pixels, and loss divergence.The authors report that latent alignment before pixel-domain optimization mitigates this instability; removing the intermediate stage or starting the final stage from random initialization also causes instability and grid-like artifacts.Source: §A.3 · p. 12
  • LUA has no explicit bound on its output latent values, so out-of-distribution inputs can still produce latents that decode to localized corruptions.The authors identify latent-domain regularization or bounded residual corrections as possible directions for improving robustness under distribution shift.Source: §A.3 · p. 12

A.4. Multi-scale Training and Resource Usage

The supplement describes shared-backbone training for ×2 and ×4 upscaling, validation-prompt construction, and baseline evaluation details. The shared multi-scale model reduces training cost relative to training separate scale-specific models.

A.4. Multi-scale Training and Resource Usage

  • Early in each stage, both heads receive supervision at every iteration; during the final 50,000 steps, one head is sampled with equal probability per iteration.The two training pairs share the same high-resolution target, and the protocol is reused across stages while the loss composition changes.Source: §A.4 · pp. 12–13
  • The shared multi-scale model supports ×2 and ×4 upscaling and saves about 2.7 × 10^2 GPU-hours versus training two separate models, roughly halving training cost.The two independent models would require approximately 5.4 × 10^2 GPU-hours, while the multi-head design yields one checkpoint for both scales.Source: §A.4 · p. 13

B. Validation Prompts and Captions

B.1. Prompt Generation Procedure

  • A GPT-based vision–language model generates one short caption per validation image using a fixed decoding configuration, and the resulting caption is reused across methods and resolutions.Source: §B.1 · p. 13

B.2. Prompt Set Statistics and Examples

  • The captions are typically one or two sentences, about 25–40 tokens, describing objects, layout, and style; examples show that high-resolution outputs can duplicate objects or produce overly dense textures.The prompt remains fixed across the displayed 1024², 2048², and 4096² generations.Source: §B.2 · p. 13

C. Baseline and Reproduction Details

C.1. Diffusion and Super-resolution Baselines

  • For FLUX evaluations, the study uses FLUX.1-Krea-dev latents generated with 12 denoising steps and decoded with the native FLUX VAE.Source: §C.1 · p. 14
  • The comparisons include direct SDXL sampling, HiDiffusion, DemoFusion, pixel-space SwinIR, and LSRNA–DemoFusion; SwinIR upsamples a 1024² SDXL image to the target resolution.Source: §C.1 · pp. 13–14

C.2. Shared Evaluation Protocol

The evaluation protocol controls shared settings so comparisons isolate the high-resolution strategy. The additional-ablation analysis examines statistical differences between latent- and pixel-space super-resolution and LUA’s behavior across those distributions.

C.2. Shared Evaluation Protocol

  • The protocol holds settings constant across methods and resolutions to make high-resolution-strategy comparisons as controlled as possible.For SDXL, runtime measures a full forward pass per image.Source: §C.2 · pp. 14–15
  • All SDXL methods use the same checkpoint, DDIM sampler, guidance scale, mixed precision, prompts, and fixed seed; runtime is mean wall-clock time per image.The seed is 42, giving deterministic sampling and identical noise realizations across baselines.Source: §C.2 · p. 15

D. Additional Ablations

  • The additional analysis complements the earlier ablations by comparing latent and pixel super-resolution statistics and assessing how LUA shifts low-resolution statistics toward high-resolution ones.Source: D. Additional Ablations · p. 15

D.1. Latent vs. Pixel Super-resolution Statistics

Low- and high-resolution distributions differ much more in latent space than in pixel space. LUA partially shifts latent statistics toward the high-resolution manifold, while the VAE decoder reduces remaining pixel-space discrepancies.

D.1. Latent vs. Pixel Super-resolution Statistics

  • Latent HR statistics are multimodal, while LR latents collapse modes; pixel LR and HR distributions retain similar shapes with small mean and scale shifts.The analysis pools per-image, channel-aggregated means and standard deviations and estimates their densities with KDE.Source: §D.1 · p. 15, Figure 10 · p. 15
  • About 40× larger, the LR–HR Wasserstein distance W1 is 0.119 in latent space versus 0.003 in pixel space.JSD and relative variance gap follow the same trend.Source: Table 5 · p. 15, §D.1 · p. 15
  • LUA moves LR latents closer to the HR manifold, but its distributions do not perfectly match HR under the downscaled-input setting.In pixel space, the VAE decoder further reduces residual discrepancies by smoothing latent errors.Source: §D.1 · p. 15

D.2. Hallucinations in High-resolution Sampling

The SDXL audit finds hallucinations become more frequent at higher native sampling resolutions, supporting generation at moderate resolution followed by upscaling. Additional FLUX examples show latent upscaling with limited runtime overhead and preserved fine detail.

D.2. Hallucinations in High-resolution Sampling

  • The audit uses a two-step zero-shot vision-language protocol to detect hallucinations and assign labels from a fixed fine-grained taxonomy.Multiple artifact labels may be assigned to an image.Source: §D.2 · pp. 15–16, Table 6 · p. 16
  • In this audit, no hallucinations were detected at 1024², while 9% of images at 2048² and 19% at 4096² contained at least one artifact.Table 6 counts flagged images out of 100 per resolution; frequent failures included Merged Objects, Incorrect Anatomy, and Merged Limbs.Source: Table 6 · p. 16, §D.2 · p. 16
  • The authors support generating near 1024² and then upscaling, rather than sampling natively at 2K–4K, where artifact frequency and diversity rise.They relate the pattern to diffusion backbones being optimized mainly at moderate resolutions and weakly exposed to very large images.Source: §D.2 · p. 16

E. Additional FLUX-based Visualizations

Loading 2511.10629v2…