Source-linked AI summary
Image Generation with a Sphere Encoder
Kaiyu Yue, Menglin Jia, Ji Hou, Tom Goldstein
TL;DR
The paper targets the high inference cost of image generation methods that require many forward passes. It introduces a Sphere Encoder that maps images to a uniformly distributed spherical latent space and decodes random points, achieving competitive one-step and state-of-the-art few-step performance across datasets, while reported FIDs trail some recent models.
Problem
Many diffusion and autoregressive image generators require many forward passes, while standard VAE prior samples can fail to decode into valid images because of the posterior hole problem.
Method
The Sphere Encoder maps natural images uniformly onto a sphere and generates images by decoding random spherical points, with iterative encoding and decoding available for few-step generation.
Results
The approach achieves competitive one-step generation and state-of-the-art performance in few-step regimes, including fewer than five steps, across several datasets.
Takeaways & Limitations
A tightly controlled spherical latent distribution enables an autoencoder that can be directly sampled for few-step, high-fidelity image synthesis.
Takeaways & Limitations
Reported FID scores trail some recent high-performance generative models, possibly because pixel-space similarity losses contribute to subtle edge blurring.
Abstract
from arXiv · showhide
We introduce the Sphere Encoder, an efficient generative framework capable of producing images in a single forward pass and competing with many-step diffusion models using fewer than five steps. Our approach works by learning an encoder that maps natural images uniformly onto a spherical latent space, and a decoder that maps random latent vectors back to the image space. Trained solely through image reconstruction losses, the model generates an image by simply decoding a random point on the sphere. Our architecture naturally supports conditional generation, and looping the encoder/decoder a few times can further enhance image quality. Across several datasets, the sphere encoder approach yields performance competitive with state of the art diffusions, but with a small fraction of the inference cost. Project page is available at https://sphere-encoder.github.io .
1. Introduction
The Sphere Encoder addresses the high cost of conventional image generation by mapping images uniformly onto a spherical latent space and decoding random points in as little as one forward pass. It supports conditional and few-step generation, with competitive one-step and state-of-the-art few-step results across datasets.
- Diffusion and autoregressive models require many forward passes, making image generation slow and costly.
- The Sphere Encoder maps natural images uniformly onto a sphere and decodes random spherical points into generated images.Its encoder-decoder design enables direct sampling from the learned latent space.
- The architecture supports conditional generation, classifier-free guidance, and few-step iteration for improving sample quality.
- The method achieves competitive one-step generation and state-of-the-art performance in few-step regimes using fewer than five steps across several datasets.
- Unlike standard VAEs, the approach aims to make random latent samples directly decodable by avoiding the posterior hole problem associated with Gaussian priors.Standard VAE prior samples may fail to produce valid images, whereas the Sphere Encoder samples random points from the sphere.
2. Method
The method maps image representations onto a spherical latent space and trains a decoder to reconstruct images from clean and noisy spherical latents. Generation samples and decodes spherical points, while iterative encoding and decoding, conditioning, and guidance support refinement and control.
- Spherical Latent Space: The Transformer encoder maps an image x into a latent z with spatial dimensions h = H/P and w = W/P and channel depth d.The patch size P determines the latent resolution.
- Spherical Latent Space: The spherifying function f flattens z and applies RMS normalization to produce a global spherical latent vector v.The paper uses simple vector RMS normalization rather than a vMF-based construction.
- Spherifying with Noise: During training, Gaussian perturbations are added to v, reprojected onto the sphere, and decoded so noisy latent clouds cover the continuous spherical space.The noise magnitude σ is jittered by sampling r uniformly from [0, 1] up to σmax.
- Generation: At inference, one-step generation decodes a spherified random vector, while few-step generation repeatedly encodes and decodes with controlled noise.The architecture also supports class conditioning, classifier-free guidance, and encoder- or decoder-space guidance operations.
- Training Objective: The objective combines pixel reconstruction, pixel consistency, and latent consistency losses to produce valid images and a smooth, semantically structured latent space.Latent consistency compares encoder representations using cosine similarity and also supports iterative generation by cleaning distorted decoder outputs.
- Architecture: The implementation uses ViT encoder and decoder components with MLP-Mixers, RMSNorm, RoPE, sinusoidal positional encoding, and conditional AdaLN-Zero modules.The paper reports that removing sinusoidal positional embeddings hurts generation quality.
3. Quantitative Experiments
Across CIFAR-10, Animal-Faces, Oxford-Flowers, and ImageNet, the sphere encoder delivers competitive generation quality with very few sampling steps. It supports strong one-step and few-step results, while iterative sampling can improve quality but may expose a trade-off between FID and perceptual realism.
- Evaluation setup: The method is evaluated with generation FID and Inception Score on datasets spanning 32 × 32 and 256 × 256 images.
- CIFAR-10: 100× fewer sampling steps than diffusion methods are used for unconditional CIFAR-10 generation while achieving better gFID and IS.
- ImageNet: At comparable parameter counts on ImageNet, the sphere encoder achieves competitive FID scores with fewer sampling steps and outperforms several established baselines.
- Metric trade-offs: FID can improve beyond 4 ImageNet sampling steps while local edges sharpen, but object structures may become more abstract and less semantically coherent.
4. Qualitative Experiments
Qualitative analyses show that the learned spherical latent space is conditionally uniform and transitions sharply between object classes. These properties reduce hybrid samples and support reliable decoding of uniformly sampled latent vectors.
- Latent interpolation: Latent interpolation produces fast transitions between object types instead of lingering in absent half-cheetah/half-dog states.
- Latent interpolation: The model’s fast transitions reduce the probability of sampling impossible or hybrid images, unlike the slow transitions observed in some GANs.
- Conditional uniformity: A conditional encoder makes each class cover the sphere uniformly, so uniformly sampled vectors can decode into the desired object class.
- Conditional uniformity: The latent visualization projects encoded CIFAR-10 samples into 3D using a random Gaussian matrix followed by unit-length normalization.
- Conditional uniformity: Conditional latent distributions achieve even coverage of the sphere for each class in isolation.
5. Image Editing
The sphere encoder supports training-free semantic manipulation and image crossover through repeated encoding and decoding. Iterative processing adapts images toward target classes or harmonizes composites while preserving source structure.
- Overview: Training-free editing uses the expressivity and robustness of the latent space for semantic manipulation and image crossover.
- Conditional manipulation: Repeated encoding and decoding of an out-of-domain image adapts its texture to a target ImageNet class while preserving primary structure.
- Conditional manipulation: Four-step iterative generation further refines class-specific characteristics and textures while maintaining the original image’s structural integrity.
- Image crossover: Processing a stitched composite for up to 10 steps harmonizes source content and smooths boundary discontinuities.
- Image crossover: The encoder directly projects stitched images into latent space without added noise, providing a deterministic path toward a coherent spherical-manifold sample.
6. Main Ablations
The ablations identify how noise geometry, loss terms, latent resolution, and sampling schedules affect Sphere Encoder generation quality. Higher angles, consistency losses, suitable latent resolutions, and shared-noise sampling improve results under different conditions.
- Noise Magnitude: As α increases toward 90°, gFID drops and generated images gain more realistic details, whereas lower α produces abstract and blurry samples.The ablation varies α from 45° to 88°, corresponding to σmax values from 1 to 28.6.
- Noise Magnitude: α = 80° works best for 32 × 32 CIFAR-10, while α = 85° works best for 256 × 256 ImageNet.The preferred angle varies slightly with image size but is otherwise dimension-invariant across latent dimensions.
- Training Loss: Adding pixel consistency loss removes the serious “waffle” artifact produced by pixel reconstruction loss alone, while latent consistency loss yields the best performance.The losses encourage consistent decoding from perturbed latents and guide the encoder-decoder pair toward a coherent latent manifold.
- Latent Spatial Resolution: Higher latent spatial resolution works best on ImageNet with compression ratio 3.0, while smaller datasets favor higher resolution with compression ratio 1.5.The finalized latent dimensions are reported in Appendix D.
- Sampling Schemes: Fixed noise strength with shared noise across all steps is the best ImageNet sampling scheme, outperforming decaying strength and independent noise.The authors hypothesize that shared noise maintains a consistent direction during the spherical optimization path.
- Uniform Latent Distribution: Explicit uniform-distribution regularization is unnecessary because the method naturally achieves the desired latent properties.This conclusion is supported by additional ablations in Appendix C.3.
7. Related Work
Prior work uses variational inference, GANs, or diffusion mechanisms for spherical latents and image generation. Sphere Encoder instead traverses between latent and pixel spaces through a spherical latent space for few-step generation.
- Spherical Latent Spaces: Spherical latent-space methods based on variational inference inherit posterior-prior mismatch and difficult high-dimensional sampling procedures.The cited approaches use priors such as the von Mises-Fisher distribution.
- Few-Step Generation: GANs provide inherently one-step generation, whereas diffusion models typically require hundreds to thousands of iterative steps.Distillation and consistency techniques accelerate diffusion generation to a few steps while approximating the original diffusion trajectory.
- Pixel-Space Generation: Pixel-space generation is common for GANs but challenging for diffusion models because of pixel-space dimensionality.Recent diffusion-based methods have nevertheless advanced pixel-space generation.
- Sphere Encoder: Sphere Encoder differs fundamentally by iteratively traversing between latent and pixel spaces, grounded in a spherical latent space.This mechanism supports its few-step generation approach.
- Sphere Encoder: The work draws inspiration from wireless-network sphere encoders and decoders that distribute codewords uniformly across a sphere.
8. Conclusion
Sphere Encoder directly samples a uniformly controlled spherical latent space to enable few-step, high-fidelity image synthesis. The paper presents this as a proof of concept with substantial room for improvement and identifies encoder overhead as a drawback.
- Conclusion: Uniformly controlling latent distributions on a sphere enables an autoencoder that can be directly sampled for few-step, high-fidelity synthesis.
- Conclusion: The implementation is presented as a proof of concept for direct conditional and unconditional generation from an autoencoder and is likely far from optimal.
- Limitations: The approach requires parameters for both encoder and decoder, plus two encoder passes during training for latent encoding and consistency loss.Future improvements could eliminate the encoder at inference time and possibly during training.
A. Additional Results on CIFAR-10
Classifier-free guidance with CFG = 1.2 slightly improves CIFAR-10 generation over the no-CFG baseline.
- Classifier-Free Guidance: With 6-step sampling, CFG = 1.2 reduces gFID from 1.65 to 1.41 and increases IS from 10.7 to 10.8.
B. Memorization Risk on CIFAR-10
Longer CIFAR-10 training can produce near-duplicate samples, indicating possible memorization, although the generated bird is transformed rather than copied exactly.
- Around 10K training epochs, the model sometimes generates near-duplicate bird samples across different sampling runs.Figure 19 illustrates near-duplicate birds corresponding to a real training image.
- The near-duplicates suggest memorization of some training samples when the model overfits the small-scale CIFAR-10 distribution.
- The generated bird is flipped relative to the real image, suggesting transformation rather than exact copying.
- Longer training nevertheless improves generation quality, reaching gFID = 0.94 and IS = 11.1 with 10-step sampling and CFG.
C. Additional Ablations
The ablations examine classifier-free guidance placement and report few-step conditional generation results on CIFAR-10, including longer-training settings.
- CFG Position: CFG can be applied after decoding in pixel space, after encoding in latent space, or in both spaces using the combo option.
- CFG Position: Pixel-space CFG consistently outperforms latent-space CFG, while combo CFG = 1.6 gives the best 4-step ImageNet result.
- CIFAR-10 Conditional Generation: Table 7 reports few-step conditional generation results on CIFAR-10 after longer 10K-epoch training.
- CIFAR-10 Conditional Generation: Table 8 reports few-step CIFAR-10 generation results using classifier-free guidance.
C.2. Dialing in the Noise Distribution
Additional ablations tune noise angles, latent regularization, normalization, compression, and noise priors, identifying settings that improve quality while exposing practical trade-offs.
- C.2. Dialing in the Noise Distribution: Mixing angles from [80°, 85°] with 0.1 probability improves one-step and four-step quality, whereas [80°, 89°] degrades quality and causes gradient explosions.
- C.2. Dialing in the Noise Distribution: The adopted angle ranges are [80°, 85°] for CIFAR-10 and [85°, 89°] for 256-resolution datasets, each mixed with probability 0.1.
- Explicit Uniform Regularization: Batch Normalization marginally improves one-step and four-step quality, while adding SWD slightly degrades it and increases complexity or cost.The authors therefore view additional BN or SWD regularization as potentially unnecessary because noisy spherifying already encourages near-uniformity.
- C.5. Volume Compression Ratio: Few-step generation has optimal pixel/latent volume compression ratios around 1.5 and 3.0, far below the typical diffusion-autoencoder ratio of 48.
- C.6. Noise Prior Distribution: Generation quality is insensitive to the choice between standard and truncated Normal noise priors, with similar gFID and IS across truncation thresholds.
- Experimental Settings: The main experiments use specified training hyperparameters and report that EMA weight smoothing may not noticeably change FID or sample quality.