Source-linked AI summary
Efficient Image Synthesis with Sphere Latent Encoder
Tung Do, Thuan Hoang Nguyen, Hao Li
TL;DR
Few-step generation methods can be unstable or difficult to scale, and Sphere Encoder adds repeated pixel–latent computation while jointly optimizing competing reconstruction and generation objectives. The paper decouples these roles with a fixed pretrained encoder and a separate spherical-latent denoiser, reporting better quality and inference efficiency than Sphere Encoder across three datasets. Its scope remains limited by class-conditional evaluation and dependence on the pretrained encoder.
Problem
Few-step models can be unstable and hard to scale, while Sphere Encoder incurs repeated pixel–latent computation and an objective conflict between reconstruction and generation.
Method
The method fixes a pretrained representation autoencoder as an image tokenizer and trains a separate denoising model entirely in spherical latent space.
Results
Across Animal-Faces, Oxford-Flowers, and ImageNet-1K, the method significantly outperforms Sphere Encoder in generation quality and inference speed while competing with other baselines.
Takeaways & Limitations
Latent-only denoising provides a simpler, more efficient pipeline while allowing reconstruction and generation components to specialize independently.
Takeaways & Limitations
The method is evaluated only for class-conditional generation and depends on a strong pretrained representation autoencoder, which may limit generality and make one-step sampling challenging.
Abstract
from arXiv · showhide
Few-step image generation has seen rapid progress, with consistency and meanflow-based methods significantly reducing the number of sampling steps. Despite their low inference cost, these approaches often suffer from training instability and limited scalability. Sphere Encoder is a recent alternative that produces high-quality images in only a few steps; however, it requires repeated transitions between the pixel space and latent space during inference while jointly optimizing reconstruction and generation within a single architecture. This design leads to computational inefficiency and objective conflict between reconstruction and generation. To address these limitations, we decouple the framework into a fixed pretrained image encoder and a separate latent denoising model trained entirely in a spherical latent space. Our approach eliminates repeated pixel-space operations during training and inference, improving efficiency and allowing reconstruction and generation to specialize independently. On Animal-Faces, Oxford-Flowers and ImageNet-1K datasets, our method significantly outperforms Sphere Encoder in both generation quality and inference speed, while achieving competitive results against strong few-step and multi-step baselines.
1 Introduction
The paper identifies inefficiencies and objective conflict in Sphere Encoder, then proposes a decoupled spherical-latent framework that improves efficiency and generation quality across several datasets.
- Motivation: Few-step generative models reduce sampling cost but can remain unstable and difficult to scale.Flow matching commonly requires iterative sampling, while recent few-step approaches target lower inference cost.
- Motivation: Sphere Encoder jointly models reconstruction and generation while producing images from Gaussian noise in only a few steps.Its encoder–decoder architecture projects latents onto a hypersphere rather than using a KL-divergence objective.
- Limitations: Repeated pixel–latent transitions during Sphere Encoder generation add computational overhead during training and inference.Each iteration decodes a latent to pixels and re-encodes the resulting image.
- Limitations: Joint reconstruction and generation optimization creates a trade-off: lower noise improves reconstruction but weakens generative performance.The paper argues that these objectives favor different representations.
- Method: The proposed method uses a fixed pretrained image tokenizer and a separate denoising model trained and sampled entirely in spherical latent space.The decoder is invoked once at the end, decoupling reconstruction from generation.
- Results: Approximately 85% lower inference cost, or about 6.5× fewer FLOPs, is reported versus Sphere Encoder at the same sampling step.Experiments cover Animal-Faces, Oxford-Flowers, and ImageNet-1K, with improvements in generation quality and inference efficiency.
2 Related Works
Related work spans few-step flow-matching, autoencoder latent-space modeling, and the reconstruction–generation trade-off that motivates decoupling.
- Efficient Image Generation: Few-step flow-matching models can achieve strong class-conditional generation but face high training costs and unstable Jacobian–vector product computations.These limitations make scaling to larger settings challenging.
- Sampling in Autoencoder Latent Spaces: Variational autoencoders typically use unimodal Gaussian posteriors, limiting their ability to capture multi-modal latent structure.The resulting mismatch can produce suboptimal inference and weak latent representations.
- Trade-off between reconstruction and generation: Increasing visual-tokenizer capacity can improve reconstruction while making generation harder and requiring larger denoisers or more training.This reflects a trade-off rather than a guaranteed improvement in generation.
- Trade-off between reconstruction and generation: Strong semantic latent representations are important for generative performance, whereas reconstruction-focused training can yield weak semantic features.Sphere Encoder similarly observes that lower noise improves reconstruction but degrades generation by reducing hyperspherical coverage.
3 Background
Sphere Encoder spherifies noisy autoencoder latents and trains with reconstruction and consistency objectives, but its iterative pixel–latent generation process is computationally inefficient.
- Spherification function and additive noise: Sphere Encoder maps images to latents, projects flattened representations onto a hypersphere, and perturbs them with two Gaussian noise levels.The encoder and decoder operate on latent tensors whose spatial dimensions derive from image size and patch size.
- Spherification function and additive noise: Noise magnitudes are sampled with σ ∼ U(0, σmax) and σsub ∼ U(0, σ × 0.5), encouraging dense hyperspherical coverage.The stated goal is to help the decoder learn a smooth mapping over the latent manifold.
- Training objectives: Pixel reconstruction loss combines ℓ1 and LPIPS losses to preserve fidelity between reconstructed images and inputs.Pixel consistency additionally compares reconstructions produced from different noise levels to promote semantic stability.
- Training objectives: Latent consistency minimizes cosine similarity loss between a re-encoded noisy reconstruction and the clean latent representation.This encourages distorted or off-manifold reconstructions to return toward stable latent representations.
- Training objectives: The total objective is a weighted combination of pixel reconstruction, pixel consistency, and latent consistency losses.The weights λ1, λ2, and λ3 control the contributions of the three terms.
- Limitations: During generation, repeated encoder–decoder transitions introduce significant overhead, while joint reconstruction and generation optimization is suboptimal.The paper motivates separate stages because better reconstruction does not necessarily improve generation.
4 Methodology
The method performs few-step generation entirely in spherical latent space, using a fixed pretrained autoencoder and a separate denoising model. Training shifts reconstruction and consistency supervision to latent space, while sampling iteratively projects, denoises, and re-noises latents before one final decode.
- 4.1 Denoising model on Spherical Latent Space: The framework uses a pretrained representation autoencoder and a transformer-based denoising network operating directly on spherical latents.The encoder maps images to structured latents, while the denoising network predicts clean latents from noisy spherical projections.
- 4.1 Denoising model on Spherical Latent Space: The model denoises corrupted latent inputs without conditioning on the noise level or timestep during training or inference.
- 4.2 Training Objective: Both reconstruction and consistency objectives are computed in latent space, reducing the pixel-space overhead associated with Sphere Encoder.The latent-space losses enable new samples to be generated in only a few steps.
- 4.2 Training Objective: Reconstruction supervision combines ℓ1 and cosine-similarity losses between predicted and clean latents, encouraging alignment for accurate reconstruction.
- 4.2 Training Objective: Consistency supervision compares predictions from two noise levels, using the lower-noise prediction as a stop-gradient target to promote smoothness across nearby spherical regions.The objective encourages nearby regions on the hypersphere to correspond to similar image content.
- 4.4 Sampling in Spherical Latent Space: Sampling begins from Gaussian noise and repeatedly projects, denoises with classifier-free guidance, re-projects, and re-noises the latent while decreasing perturbation magnitude.The decoder is invoked once at the end to map the final latent back to pixels.
5 Experiments
Experiments evaluate generation quality, computational cost, and design choices across Animal-Faces, Oxford-Flowers, ImageNet-1K, and ImageNet-100. The method improves quality and efficiency over Sphere Encoder, with ablations identifying effective noise and training objectives.
- Few-step Image Generation: The method significantly outperforms Sphere Encoder on small datasets while using a smaller latent-space sampling architecture with substantially lower GFLOPs.The comparison includes 2, 4, and 6 sampling steps, with variants using and omitting classifier-free guidance.
- Few-step Image Generation: FID decreases from 4.02 to 2.25 at the same 4 × 2 NFE budget, reaching 2.11 at 6×2 NFE on ImageNet-1K.The method produces sharper images with finer details than Sphere Encoder, while multi-step baselines require orders of magnitude more evaluations.
- Few-step Image Generation: The method remains substantially more efficient than Sphere Encoder even when using more sampling steps.Its low-NFE performance is comparable to recent one-step diffusion and flow-based approaches, while avoiding Jacobian-vector-product approximations.
- Ablation Studies: Uniform, LogNorm(−0.4, 1.0), and LogNorm(+0.4, 1.0) noise distributions each improve FID by at least 10.0% over the baseline.LogNorm(−0.4, 1.0) yields a 13.5% improvement, while LogNorm(+0.4, 1.0) performs best with a 17.4% gain and is used thereafter.
- Ablation Studies: Combining reconstruction and consistency losses achieves the best ablation result, with a 47.8% improvement over the reference setting.Consistency loss alone improves performance by 40.8%, whereas adding latent consistency produces a smaller 46.3% improvement.
- Ablation Studies: Increasing inference steps improves fidelity, while stronger semantic representations and larger latent dimensionality yield sharper samples.These factors are analyzed in Figure 5 as separate effects on sample quality.
6 Limitations and Conclusion
The method operates entirely in spherical latent space and decouples reconstruction from generation, but its scope remains limited by class-conditional evaluation and dependence on a fixed pretrained encoder.
- Limitations: The method focuses on class-conditional image generation and does not evaluate text-to-image generation.
- Limitations: Performance may vary with the choice and quality of the pretrained encoder, creating generality, computational, and availability constraints.
- Limitations: High-quality one-step sampling remains challenging because the encoder is fixed.
- Conclusion: The framework operates entirely in spherical latent space and avoids repeated pixel–latent transitions through decoupled reconstruction and generation.
A Implementation
The implementation uses a pretrained representation autoencoder to produce fixed latents, then trains denoising models on 256 × 256 data across three datasets with limited augmentation.
- Denoising model architecture: The denoising model uses a transformer-based SiT architecture, with SiT-B/1 used for Animal-Faces and Oxford Flowers.
- Representation autoencoder: The representation autoencoder maps 256 × 256 × 3 images to 16 × 16 × 768 latents using a DINOv2-B encoder and ViT-XL decoder.
- Representation autoencoder: The autoencoder contains 502M parameters, requires 128 GFLOPs per forward pass, and has reported reconstruction FID of 0.49.
- Datasets and training details: The experiments use Animal-Faces, Oxford Flowers, and ImageNet-1K at 256 × 256 resolution.
- Datasets and training details: Images are pre-encoded with DINOv2-B, and horizontal flipping with probability 0.5 is the only listed augmentation.
B Additional Technical Details
Additional details describe latent consistency training, evaluation protocols, FLOP accounting, and progressively decayed noise perturbation during sampling.
- Latent Consistency Loss: The latent consistency procedure refines a noisy latent, projects it onto the hypersphere, re-noises it, and enforces cosine consistency between predictions.
- Experiment with latent consistency loss: Including latent consistency loss from training start increased iteration time from 201 ms to 336 ms and slowed convergence.
- Metric Calculation: Evaluation generates 50,000 class-balanced images and uses dataset-specific reference distributions, with Guided Diffusion statistics for ImageNet-1K.
- FLOP Calculation: Sampling FLOPs are computed per image at batch size 1 from denoising-model passes plus one final decoder pass.
- Noise perturbation during sampling: A progressively decaying injected-noise schedule generally improves performance, with γ = 0.5 used for ImageNet-1K.
C Additional Quantitative Results
Additional ImageNet-1K analyses examine sampling-step saturation, computational cost, component complexity, and performance on CMMD alongside FID and other metrics.
- Few-step Image Generation on ImageNet-1K: ImageNet-1K generation quality improves substantially from 2 to 4 steps, then improves marginally beyond 4 and nearly saturates at 6 steps.
- Computational cost: At comparable sampling budgets, the method is evaluated against Sphere Encoder using total GFLOPs.
- Performance on Additional Metrics: The method achieves competitive CMMD performance, outperforming several multi-step flow-matching methods and the one-step Improved MeanFlow model.
- Performance on Additional Metrics: Table 6 reports FID and CMMD as lower-is-better metrics, while IS, Precision, and Recall are higher-is-better metrics.
D Additional Qualitative Results
Additional qualitative results show 256 × 256 samples from the best-performing model, with 6-step sampling producing sharper images and fewer distortions than 4-step sampling.
- 256 × 256 samples were generated using SiT-XL/1 with DINOv2-B, classifier-free guidance of 3.2, σmax = 24, and γ = 0.5.
- 6-step sampling consistently produces sharper samples with reduced distortions compared with 4-step sampling.
E Broader Societal Impacts
The method improves image-generation efficiency and supports lower computational cost, while easier production of realistic images may increase misuse risks.
- The method improves image-generation efficiency, enabling content-creation and machine-learning applications while reducing computational cost.
- Easier generation of realistic images may increase misuse risks, including deceptive or misleading content.
- The paper encourages safeguards such as detection methods and usage guidelines.