Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion

1. Introduction

• The VAE’s texture-focused representation leaves the diffusion model to learn high-level semantics and low-level detail together, contributing to slow convergence and suboptimal generation quality.Source: §1 · p. 2
• Prior semantic-prior methods denoise semantics and textures synchronously, despite diffusion’s coarse-to-fine progression from low-frequency structure to high-frequency texture.SFD instead proposes that semantics evolve earlier and guide texture generation.Source: §1 · p. 2
• A hard sequential schedule risks training–inference mismatch because inference must use the model’s own imperfect semantic predictions rather than ground-truth conditions.SFD’s asynchronous schedule keeps both latent types denoising simultaneously at different noise levels while semantics lead.Source: §1 · p. 2
• SFD combines compact semantic latents from a dedicated Semantic VAE with texture latents, then denoises them asynchronously in three stages.First semantics establish a coarse layout; next, semantics and textures denoise at different noise levels; finally, texture alone refines details, and the image is decoded from the texture latent.Source: §1 · p. 2
• FID 1.04 on ImageNet 256×256 and 100× faster training convergence than DiT are among SFD’s reported results.The paper also reports 33.3× faster convergence than LightningDiT and improved performance when integrating SFD into ReDi and VA-VAE.Source: §1 · p. 2

2.1. Diffusion Models for Image Generation

• SFD makes diffusion’s coarse-to-fine evolution explicit: semantic components denoise earlier and guide refinement of low-level textures at cleaner noise levels.Source: §2.1 · pp. 2–3

2.2. Semantic Representation Enhanced Diffusion

• Semantic-enhanced methods align pretrained-encoder features with diffusion features or VAE latents, jointly model semantic and texture features, or replace conventional VAE latents with encoder representations.SFD extends this line of work with asynchronous schedules that establish semantics earlier and guide texture refinement.Source: §2.2 · p. 3

2.3. Asynchronous Denoising Methods

• Unlike prior asynchronous methods that assign distinct noise schedules to tokens or pixels, SFD applies asynchronous denoising to semantic and texture subspaces within one latent representation.Source: §2.3 · p. 3

3.1. Preliminaries

• Flow matching linearly interpolates clean data and Gaussian noise; generation integrates the learned velocity field from noise toward data.The time variable ranges from 0 to 1.Source: §3.1 · p. 3

3.2.1. Semantic VAE

• SemVAE compresses frozen vision-foundation-model patch features into compact semantic latents while preserving spatial layout and minimizing information loss.Its encoder uses four Transformer blocks.Source: §3.2.1 · p. 4

3.2.2. Latent Construction

• The composite latent concatenates SemVAE-encoded semantics and texture-VAE latents along the channel dimension.The texture branch uses SD-VAE.Source: §3.2.2 · p. 4

3.3. Semantic-First Diffusion

• Semantics and textures receive separate timesteps, and the Transformer jointly predicts their velocities from the noisy composite latent and class label.The model is trained with sampled semantic and texture priors and a weighting hyperparameter.Source: §3.3 · pp. 4–5
• Training combines velocity-prediction losses for both latent types with REPA alignment to pretrained visual-encoder representations.The aligned encoder output is the same representation used as SemVAE input.Source: §3.3 · p. 5
• SFD uses three asynchronous phases: semantic-only initialization, joint denoising with semantics ahead, then texture-only completion after semantics finish.Separate timestep offsets keep semantics less corrupted; masks gate phase-specific updates, and the total inference-step count stays fixed.Source: §3.3 · pp. 4–5

4.1. Experimental Setup

• Experiments use SD-VAE texture latents and SemVAE semantic latents, with evaluation on 50K generated samples using the ADM pipeline.The setup also specifies ImageNet-1K training and ∆t = 0.3.Source: §4.1 · pp. 5–6

4.2. Main Results

• FID is optimal at 3.03 when the temporal offset is ∆t = 0.3; larger offsets degrade performance, and ∆t = 1.0 is teacher-forcing sequential generation.The figure identifies the moderate offset as yielding the lowest FID.Source: Figure 5 · p. 6, §4.2 · p. 6
• FID 1.06 with LightningDiT-XL and 1.04 with LightningDiT-XXL are SFD’s guided ImageNet 256×256 results after 800 epochs.The system-level comparison reports state-of-the-art performance with guidance.Source: §4.2 · p. 6, Table 2 · p. 7
• At 400K iterations, SFD reaches FID 3.53 with LightningDiT-XL/1, 2.31 points below REPA at 4M iterations and 6.09 below DiT-XL/2 at 7M.SFD reaches comparable performance to DiT-XL at 7M iterations in 70K iterations, or 100× faster convergence.Source: §4.2 · p. 6, Table 1 · p. 7

4.3. Ablation Studies

• FID falls from 8.17 to 7.08 with REPA, to 5.24 after adding SemVAE latents, and to 3.03 after adding semantic-first denoising.The component ablation reports each stage’s FID.Source: §4.3 · p. 7, Table 3 · p. 8
• SemVAE achieves FID 3.03 versus PCA’s 4.06 in the semantic-latent compression comparison.The authors attribute the difference to preserving semantic information completeness.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Generalization of Semantic-First Mechanism

• Adding semantic-first denoising to ReDi improves FID from 5.33 to 4.41.This experiment tests the mechanism in a method that composes semantic and texture latents.Source: §4.4 · p. 8, Table 5 · p. 8

4.5. Reconstruction Performance Analysis

• SD-VAE has the best reconstruction scores: rFID 0.26, LPIPS 0.089, PSNR 28.59, and SSIM 0.80.RAE loses fine details such as the “40g” text and exhibits color distortion; VA-VAE slightly compromises pixel-level reconstruction fidelity.Source: §4.5 · p. 8, Table 6 · p. 8, Figure 7 · p. 8

• SFD performs asynchronous denoising on semantic and texture latents in latent diffusion models.Source: §5 · p. 8
• On ImageNet class-conditional generation, SFD consistently outperforms competing methods in convergence speed and generation quality.Source: §5 · p. 8
• The authors identify control of the relative denoising pace between semantics and textures as crucial for efficient generative modeling.Source: §5 · p. 8

A.1. SemVAE Configuration

• SemVAE uses DINOv2-B with registers on 256 × 256 images and has four Transformer blocks in each of its encoder and decoder, with 29M parameters each.It is trained on ImageNet-1K for 1M iterations with random cropping.Source: §A.1 · p. 11

A.2. Diffusion Model Configuration

• SD-VAE encodes texture into 32 channels and SemVAE extracts 16-channel semantic representations, forming a unified 48-channel, 256-token latent per 256×256 image.The diffusion model is trained on ImageNet-1K for 800 epochs.Source: §A.2 · p. 11

A.3. Dual Timestep Embedding

• SFD uses two independent timestep embedders, each with hidden dimension H/2, for semantic and texture timesteps; their outputs are concatenated and injected into the backbone.The two embeddings provide distinct timestep signals to the semantic and texture latents.Source: §A.3 · p. 11

• AutoGuidance uses a weaker version of the main diffusion model to guide denoising, and the experiments use a degraded LightningDiT-B model as the guiding network.Source: §A.4 · p. 11
• AutoGuidance is the primary guidance method, using a weaker model to guide the main diffusion model rather than an unconditional model as in CFG.Source: §A.4 · p. 11
• For FID-50K evaluation, class-balanced sampling generates 50 images per class, for 50,000 images total.The authors adopt this protocol to align with prior work.Source: §A.4 · p. 12

B.1. Complete Comparisons

• FID 1.06 for SFD-XL (675M) beats RAE DiTDH (839M) at 1.13, and SFD-XXL (1.0B) reaches FID 1.04 on guided ImageNet 256 × 256.SFD also achieves sFID 3.75, which the authors associate with structural coherence and spatial alignment.Source: §B.1 · p. 12
• Unguided SFD remains competitive but has difficulty converging its texture latents, which are twice the dimensionality of the f8d4 VAE latents used by ReDi and REG.SFD uses the f16d32 variant, following LightningDiT.Source: §B.1 · p. 12

B.2. Inference Strategies

• At 100 unguided steps, SFD reaches FID 6.35; at 250 steps, its FID is 6.32, while other baselines need 200–250 steps to approach optimal FID.The authors suggest that stabilizing global semantics early may reduce the refinement steps needed for high-quality sampling.Source: Table 11 · p. 13, §B.2 · p. 13
• At 100 guided Euler steps, SFD (XL) achieves FID 1.045, versus 1.064 with dopri5; at 25 steps, it records 1.865 versus SVG’s 1.920.The 100-step comparison is for SFD (XL); the 25-step comparison is against SVG [40].Source: Table 12 · p. 14, §B.2 · p. 14
• SFD (XL) demonstrates consistent superiority in FID and sFID under both class-balanced and class-random sampling protocols.The comparison adopts a shared class-balanced protocol for prior methods and includes independent evaluations of VA-VAE, ReDi, and REG.Source: Table 13 · p. 14, §B.2 · p. 14

B.3. Unconditional Generation

• With AutoGuidance, unconditional SFD reaches FID 3.77 at 80 epochs and 2.90 at 200 epochs; without it, FID falls from 10.24 at 80 epochs to 8.46 at 200.The authors attribute the gains to asynchronous denoising and suggest smoother semantic representations provide global structural cues without class-label guidance.Source: §B.3 · p. 14, Table 14 · p. 14

B.4. SFD for VA-VAE

• SFD improves VA-VAE from FID 4.52 to 4.14, but remains behind SD-VAE-based SFD at FID 3.03.The authors suggest that VA-VAE’s entangled semantic and texture latent space limits the flexibility of asynchronous denoising.Source: §B.4 · pp. 14–15, Table 15 · p. 15

B.5. Computational Cost

• At 400K iterations, SFD adds less than 0.01% FLOPs while substantially improving generation quality.The implementation adds a 16-channel semantic latent and replaces one timestep embedder with two; the authors report the total parameter count is smaller.Source: Table 16 · p. 15, §B.5 · p. 15

C.1. Semantic VAE Design

• DINOv2-B gives the lowest FID among tested target encoders at 3.03, while DINOv2-L gives the best result across DINOv2 scales at 2.97.The authors select DINOv2-B as the default encoder for its performance-efficiency trade-off.Source: §C.1 · p. 15
• Increasing semantic latent capacity from 2 to 16 channels consistently improves performance, with 16 channels achieving FID 3.03.The study uses DINOv2-B features, which have 768 dimensions before compression.Source: §C.1 · p. 15

C.2. Effect of semantic loss weight

• β = 2.0 yields the lowest FID of 3.03; larger weights of 4.0 or 8.0 degrade performance, which the authors associate with suppressed texture learning and loss of fine details.FID decreases as β rises from 0.25 to 2.0.Source: §C.2 · pp. 15–16, Table 18 · p. 16

C.3. Effect of REPA configurations

• REPA alignment at depth 2 gives FID 3.03, whereas the original REPA [50] reports its optimum at depth 8.The authors explain that their REPA loss decodes semantic representations from noisy compressed latents, unlike the original method’s distillation role.Source: Table 19 · p. 16, §C.3 · p. 16
• The lowest FID occurs at REPA loss weight λ = 0.5, which the authors describe as balancing semantic consistency and generative fidelity.Source: Table 19 · p. 16, §C.3 · p. 16
• The combined cosine+MSE REPA objective achieves the best reported performance at FID 3.03, outperforming single-loss variants.The authors describe MSE as providing distribution-level precision and cosine similarity as enhancing directional alignment.Source: Table 19 · p. 16, §C.3 · p. 16
• The final model uses Table 8 settings rather than the ablation optima because the ablations ran for 400K iterations, while the full training duration was 4M.Source: §C.3 · p. 16

D. Limitation and Future Work

• SFD uses a fixed temporal offset, which may not suit all noise levels; future work could study dynamic or adaptive schedules.The authors also identify eliminating auxiliary REPA feature-alignment supervision as a future direction.Source: D. Limitation and Future Work · pp. 16–17
• Extending SFD to text-to-image and text-to-video generation is proposed as future work to examine multimodal guidance and temporal consistency.Source: D. Limitation and Future Work · p. 17

E. More Visualization Results

• Using the same initial noise, SFD forms clearer structures and more realistic details earlier than LightningDiT, REPA, and VA-VAE variants.The comparison concerns early training stages and indicates faster convergence.Source: E. More Visualization Results · p. 17
• Further SFD visualizations show bald eagles, castles, lakesides, and hot-air balloons.Source: E. More Visualization Results · p. 17, Figure 9 · p. 18, Figure 15 · p. 20
• Additional SFD visualizations cover cockatoos, pandas, teapots, hamburgers, and strawberries.Source: Figure 10 · p. 18, Figure 11 · p. 18, Figure 12 · p. 19