PaperlayerView exact revision on arXiv

Source-linked AI summary

UltraDiff: Transferring High-Fidelity Priors to Compressed Latent Spaces for High-Resolution Image Generation

Jingjing Ren, Haitian Zheng, Haoyu Chen, Connelly Barnes, Zhifei Zhang, Yuqian Zhou, Zhe Lin, Lei Zhu

arXiv:2610.21202v1cs.CV

TL;DR

High-resolution synthesis must balance computational cost with fidelity: f32 compression is efficient but leaves an ambiguous target that can hinder detail and layout. UltraDiff combines joint f8–f32 learning with f32-space score distillation; it approaches evaluated f8 quality while retaining f32 efficiency, with a reported 2K generation time of 6.2 seconds.

  • Problem

    f32 compression reduces token count for efficient synthesis, but its ambiguous target makes fine detail and precise spatial layout harder to model than with f8.

    Source: §1 · pp. 1–2

  • Method

    UltraDiff trains MixTeacher on paired f8 and f32 latents with shared attention, then distills the marginal vector field directly into the f32 space.

    Source: §1 · p. 3

  • Results

    A 2K image takes 6.2 seconds with 20 sampling steps on one A100, 40.3× faster than the f8 anchor, while UltraDiff approaches evaluated f8-system fidelity and scales to 4K.

    Source: §1 · p. 3

  • Takeaways & Limitations

    In the evaluated DiT pipeline, UltraDiff substantially narrows the f8–f32 quality gap while retaining f32 inference efficiency.

    Source: §5 · p. 10

    The training data and f8 anchor are proprietary, limiting exact reproduction of the reported checkpoint; validation with a fully open-source teacher–student pipeline remains future work.

    Source: A Limitations and Future Work · p. 14

Abstract

from arXiv · show

High-resolution synthesis with Latent Diffusion Models (LDMs) faces a fundamental trade-off between computational cost and generative fidelity. This trade off is largely governed by the latent compression factor f (e.g. f8 denotes f=8 and f32 f=32). While aggressive compression (f32) significantly improves efficiency, it suffers from severe information loss and ambiguity compared to standard f8 models. We propose UltraDiff, a framework that reformulates this challenge as joint latent diffusion learning followed by marginal score distillation. We introduce MixTeacher, a dual branch DiT that models the joint f8-f32 distribution, utilizing shared attention to resolve the ambiguity of the compressed space with high fidelity f8 priors. Crucially, we propose vector field distillation in the native f32 space. Theoretically, this compels the student to learn the conditional expectation of the teacher trajectory, effectively filtering out compression artifacts. Across the evaluated 1K to 4K settings, UltraDiff approaches f8 level quality while preserving f32 efficiency. With 20 sampling steps on a single A100 GPU, it generates a 2K image in 6.2 seconds over 40 times faster than our f8 baseline.

1 Introduction

Ultra-high-resolution generation is costly because attention scales with token count, while f32 compression reduces tokens but makes fine detail and layout harder to model. UltraDiff transfers f8 knowledge through joint-space learning and f32-space distillation, approaching f8 fidelity with f32 efficiency.

1 Introduction

  • Attention scales quadratically with token count, while f32 compression reduces the 2D latent token count by 16× relative to f8.Despite the efficiency gain, f32 makes fine-grained detail and precise spatial layout harder to capture, degrading perceptual fidelity and prompt adherence versus f8 models.Source: §1 · p. 1
  • f8 initialization consistently improves f32 performance, but its benefit quickly saturates and leaves a substantial gap, so static initialization alone is insufficient.The introduction motivates dynamic, state-dependent guidance rather than relying only on initialization.Source: §1 · pp. 1–2
  • UltraDiff learns a joint f8–f32 distribution with MixTeacher's shared attention, then distills the marginal vector field directly into the native f32 space.The student learns the conditional expectation of the teacher trajectory, integrating out ambiguity in compressed generation.Source: §1 · p. 3
  • A 2K image takes 6.2 seconds with 20 steps on one A100, 40.3× faster than the f8 anchor; at 4K, the f8 anchor runs out of memory.UltraDiff approaches the fidelity of evaluated f8 systems and produces detail-rich 4K images with strong semantic alignment.Source: §1 · p. 3

2 Related Work

Related work connects latent compression and diffusion distillation to efficient generation. UltraDiff transfers an f8 prior to f32 through initialization and cross-latent supervision, while treating compatible autoencoders as a complementary direction.

2.1 Diffusion Transformer

  • More aggressive f32 compression improves high-resolution efficiency, but the smaller spatial grid makes structure and semantic alignment harder to learn.UltraDiff transfers knowledge from f8 to f32 through initialization and cross-latent distillation, followed by mixed-resolution training.Source: §2.1 · p. 4
  • Prior work explores compatible latent spaces during VAE training, whereas UltraDiff keeps independently trained f8 and f32 VAEs fixed and learns their relationship at the diffusion vector-field level.MixTeacher uses shared attention for this cross-space transfer.Source: §2.1 · p. 4

2.2 Diffusion Distillation

  • Diffusion distillation has supported faster sampling, architectural changes, modality transfer, and feature alignment; UltraDiff applies distillation to a latent-scale mismatch between f8 and f32.Its f8 teacher guides an f32 student through initialization and cross-latent supervision.Source: §2.2 · p. 4

3 Method

UltraDiff trains a joint f8–f32 teacher and distills its vector field into an efficient f32 student. A pilot study finds that f8 initialization transfers priors, but leaves a performance gap.

3 Method

  • UltraDiff frames efficient, high-fidelity synthesis as joint-latent diffusion learning followed by marginal score distillation into the f32 space.The final student operates in the high-compression space while inheriting the high-fidelity joint distribution’s generative vector field.Source: §3 · pp. 4–5
  • A pilot study finds that f8 initialization improves an f32 model over random initialization, but the benefit saturates and leaves a performance gap.This motivates dynamic, interaction-based transfer through joint latent modeling.Source: §3 · p. 5

3.1 Joint-Latent Modeling via MixTeacher

  • MixTeacher models the joint f8–f32 distribution with separate branch parameters and shared attention, enabling cross-latent interaction.Stochastic joint, f8-only, and f32-only routing preserves each branch’s generation capability and prevents f32 over-reliance on f8 features.Source: §3.1 · p. 5

3.2 Distilling the Marginal Score

  • Minimizing expected teacher–student discrepancy makes the f32 student approximate the teacher’s conditional expected vector field, filtering ambiguity and aliasing artifacts.The paper describes the projection as a structural filter that guides the student toward a consistent manifold.Source: §3.2 · p. 6
  • 0.72 GenEval: UltraDiff outperforms the f32 model initialized from f8 parameters (0.61 GenEval), approaching the 0.73 f8 anchor.Both f32 models use the same f8 initialization, f32 VAE, and DiT backbone.Source: §3.2 · p. 6

4 Experiment

The experiments evaluate UltraDiff’s training setup, distillation and routing choices, and high-resolution performance. Controlled results report quality gains over f32 controls, while comparative and efficiency evaluations cover 2K–4K synthesis.

4.1 Implementation Details

  • F8 initialization consistently outperforms scratch training in f32, but leaves a gap to the f8 model, motivating distillation; evaluations include GenEval, CLIP, and ImageReward.The implementation trains the f32 model for 120k iterations and uses the same f32 VAE and DiT backbone across f32 variants.Source: §4.1 · p. 6, Table 1 · p. 6, §4.2 · p. 7

4.2 Ablation Study

  • The adopted routing probabilities are pjoint = 0.4, pf32 = 0.4, and pf8 = 0.2; joint-only training leads to f32 shortcut exploitation and representation collapse.Mixing joint and f8-Solo modes without f32-Solo leaves the f32 branch under-trained, so preserving independent branch capability is treated as a prerequisite.Source: §4.2 · p. 7, Table 5 · p. 8
  • In mixed inference, the f32 branch scores 0.74 GenEval versus 0.62 for f32-Solo and produces images visually indistinguishable from the f8 branch.The mixed-mode f32 output provides a structurally determinate signal for distillation in the student’s native space.Source: §4.2 · pp. 7–8, Table 3 · p. 7
  • MixTeacher’s f32-native v-prediction avoids cross-grid interpolation and yields sharper details and better text–image consistency than the evaluated feature-alignment approaches.DINOv2 feature guidance brings little gain, while f8-grid alignment also depends on learned upsampling across mismatched token layouts.Source: §4.2 · p. 8, Table 4 · p. 8
  • At 1K, UltraDiff improves DPG, GenEval, and HPSv3 by 11.38, 0.078, and 1.71, respectively, over a same-backbone control without distillation.At 4K, it also achieves better FID, aesthetic score, and CLIP similarity; the comparison uses the same f8-initialized f32 backbone and scaling stage.Source: §4.2 · p. 8, Table 6 · p. 8

4.3 Comparative Study

  • On 1024², UltraDiff ranks at or near the top among similarly sized models on DPG and GenEval and remains competitive on HPSv3.These cross-system results are deployment-oriented context, not a controlled causal comparison, because the systems differ in multiple components and settings.Source: §4.3 · p. 9, Table 8 · p. 10
  • At 2K–4K, UltraDiff reports stronger overall quality and semantic-alignment metrics than Sana; 2K sampling takes 6.2s, while the f8 model runs out of memory at 4K.The reported system comparison does not isolate any single component, and both systems are efficient at high resolution.Source: Table 7 · p. 9, §4.3 · p. 10

5 Conclusion

UltraDiff transfers high-fidelity priors into compressed f32 latents, narrowing the f8–f32 quality gap in the evaluated DiT pipeline. It generates a 2K image in 6.2 seconds and remains practical at 4K, where the f8 anchor runs out of memory.

5 Conclusion

  • With 20 sampling steps on one A100, UltraDiff generates a 2K image in 6.2 seconds, a 40.3× speedup over the f8 anchor, and completes 4K generation where the anchor runs out of memory.The conclusion characterizes the quality gap between f8 and f32 as substantially narrowed within the evaluated DiT pipeline.Source: §5 · p. 10, Table 9 · p. 10
  • The paper notes that efficient, high-quality 2K–4K generation may reduce computational and energy costs, while warning that the capability may also facilitate misleading or harmful content.It recommends safeguards including safety evaluation, watermarking, and provenance tracking.Source: §5 · p. 10

High-resolution Image Generation

The supplementary material points readers to implementation and reproducibility details, architectural specifications, and additional visual results.

High-resolution Image Generation

  • Additional high-resolution visual comparisons and a results gallery are supplied in separate supplementary files.The paper attributes this separate-file arrangement to the PDF submission’s file-size limit.Source: High-resolution Image Generation · p. 14
  • The supplement identifies sections on limitations and future work, reproducibility, data and model disclosure, architecture, and hyperparameters.It also describes training protocols and an open-source plan.Source: High-resolution Image Generation · p. 14

A Limitations and Future Work

The evidence is bounded by a single evaluated autoencoder–DiT pipeline and reliance on pretrained components, fixed VAEs, and proprietary resources. Training proceeds through f32 initialization, teacher preparation, and staged distillation and scaling, while inference uses only the final f32 student.

A Limitations and Future Work

  • UltraDiff is evaluated on one autoencoder–DiT pipeline; substantially different denoising architectures remain untested.Compatibility with other DiT-based pipelines is described as in-principle, not empirical validation.Source: A Limitations and Future Work · p. 14
  • The method assumes a strong pretrained f8 anchor and fixed f8/f32 VAEs, and its multi-stage adaptation adds training complexity.The combined adaptation cost is lower than directly training the f32 baseline, and inference uses only the standalone f32 student.Source: A Limitations and Future Work · p. 14, §B.2 · p. 15
  • Because the independently trained f8 and f32 VAEs remain frozen, MixTeacher learns their relationship at the diffusion vector-field level rather than enforcing latent-space compatibility.Compatible-VAE training is suggested as a possible way to further improve transfer.Source: A Limitations and Future Work · p. 14
  • Proprietary training data and the f8 anchor limit exact reproduction of the reported checkpoint, although the paper provides implementation and training details for evaluation with alternatives.Validation using a fully open-source teacher–student pipeline remains future work.Source: A Limitations and Future Work · p. 14

B.1 Training Strategy

  • The f32 phase-one checkpoint is initialized from the f8 model, then used to initialize the student for distillation and resolution scaling.The student trains at 512 and 1024 resolutions for 50,000 iterations, then at mixed 1K–4K resolutions for 10,000 iterations.Source: §B.1 · pp. 14–15
  • Teacher-based distillation is used only at 512–1K; at 2K–4K, the student continues with standard flow matching.The teacher is prepared at 512 and 1024 resolutions; the paper gives avoiding unstable or prohibitively expensive high-resolution teacher supervision as the design rationale.Source: §B.1 · p. 15

B.3 Model, Data, and Open-source Plan

The pipeline uses an internal proprietary f8 text-to-image model and proprietary training data, with scale figures presented as context rather than a controlled data-efficiency comparison. The authors plan to release code, but exact pretrained weights may not be releasable; experiments validate one DiT family.

B.3 Model, Data, and Open-source Plan

  • The training data and f8 anchor are proprietary, limiting exact reproduction, and the exact pretrained weights may not be releasable.The authors plan to release training and inference code, including MixTeacher and marginal score distillation.Source: §B.3 · p. 15
  • Reported image-corpus figures for other systems provide scale context, not a controlled data-efficiency comparison, because filtering, repeated sampling, and training stages differ.Several systems do not disclose directly comparable unique-image counts in their primary reports.Source: §B.3 · p. 15
  • The present experiments validate only one DiT family, although the framework can in principle use open-source components without changing the student’s inference architecture.Empirical validation on fully open-source and substantially different backbones remains future work.Source: §B.3 · p. 15

C Detailed Model Architectures and Hyperparameters

C.1 Diffusion Transformer (DiT) Architecture

The teacher and student share a DiT backbone that processes text embeddings and image latents as one concatenated sequence. The f32 latent uses patch size 1, and the backbone uses RoPE and QK-Norm.

C.1 Diffusion Transformer (DiT) Architecture

  • The shared DiT backbone processes T5-XXL text embeddings and image latents as a concatenated sequence, using patch size 1 for image embeddings.The teacher has dual branches and the student a single branch.Source: §C.1 · p. 16
  • The DiT backbone follows a standard Transformer design with Rotary Positional Embeddings and QK-Norm.Table 10 summarizes its architecture specifications.Source: Table 10 · p. 16

C.2 VAE Architectures and Training Details

UltraDiff uses two continuous, KL-regularized VAEs that are trained separately and frozen during diffusion training; the f32 VAE has lower reconstruction fidelity than the f8 VAE. Reconstruction quality does not fully determine downstream generation quality, and latent diffusability also matters.

C.2 VAE Architectures and Training Details

  • Both VAEs are continuous, KL-regularized models trained from scratch and frozen during diffusion training; their latent scales and channels refer to native VAE outputs.The models are compared with public autoencoders under the same held-out reconstruction protocol.Source: §C.2 · p. 16, Table 11 · p. 17
  • The f32 VAE has lower reconstruction fidelity than the f8 VAE, while HunyuanImage-2.1 has higher PSNR and SSIM but worse LPIPS, rFID, and PatchFID than the f32 VAE.Reconstruction quality does not fully determine downstream generation quality; latent diffusability also matters.Source: §C.2 · p. 16
  • With the same f32 VAE, GenEval rises from 0.61 with f8 initialization to 0.72 with UltraDiff, reflecting better modeling rather than changed autoencoder reconstruction quality.All f32 variants in Table 1 use the same f32 VAE.Source: §C.2 · p. 16
  • The f8 VAE downsamples by 8×, uses 16 latent channels, and employs PixelShuffle decoder upsampling.Its encoder uses channel multipliers [1, 2, 4, 6] with a base channel width of 128.Source: §C.2 · p. 17
  • The f32 VAE downsamples by 32× and uses a 64-dimensional latent, with a deep encoder and Nearest+Conv decoder upsampling.The encoder uses six stages; the decoder is designed for reconstruction from a highly compressed latent grid.Source: §C.2 · p. 17

C.3 Latent Normalization

  • Channel-wise normalization is applied before latents from the independently trained f8 and f32 VAEs are fed into the DiT.This is used to address distribution shift between the continuous latent spaces.Source: §C.3 · p. 17

C.4 Training Hyperparameters

The appendix identifies the optimizer and learning-rate schedule and points to a table of common settings and stage-specific training lengths. The checklist records reproducibility and experimental-detail answers alongside several “No” responses.

C.4 Training Hyperparameters

  • The training uses AdamW with linear warmup and cosine decay; Table 13 covers common optimizer settings and stage-specific training lengths.The passage does not give the table’s specific values.Source: §C.4 · p. 18, Table 13 · p. 18

NeurIPS Paper Checklist

Related papers

Loading 2610.21202v1…