PaperlayerView exact revision on arXiv

Source-linked AI summary

Sphere Encoder 2

Kaiyu Yue, Sean McLeish, Ruchit Rawal, Brian Bartoldson, Menglin Jia, Tom Goldstein

arXiv:2610.02208v1cs.CV

TL;DR

The original Sphere Encoder leaves out the equatorial region used by random sphere samples and produces blurry one-step generations. Sphere Encoder 2 changes the noise parameterization and separates reconstruction from generation, using latent score matching for high-noise generation. It achieves the best FDr6 at 256 × 256 and second best at 512 × 512 among pixel-space ImageNet generators, with 23× to 40× less sampling compute than pixel diffusion models.

  • Problem

    The original Sphere Encoder’s training rotation stops short of the equator used by random latent samples, and pixel-wise reconstruction loss encourages blurry generations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    Sphere Encoder 2 robustly reaches 90◦ noise, keeps pixel-wise loss for low-noise reconstruction, and uses latent score matching without pixel loss for high-noise generation.

    Source: §1 · p. 2

  • Results

    Sphere Encoder 2 achieves the best FDr6 at 256 × 256 and second best at 512 × 512 among pixel-space ImageNet generators, with 23× to 40× less sampling compute than pixel diffusion models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sphere Encoder 2 adds a complementary autoencoder-based path to one/few-step generation and could be a viable candidate for that setting.

    Source: §6 · p. 12

    Lscore matches real and fake distributions only in ConvNeXt V2-N features, and this match does not carry over to the old Inception space, contributing to higher gFID than other models.

    Source: §5 · p. 10

Abstract

from arXiv · show

Sphere Encoder is an autoencoder that generates images by decoding random points from a high-dimensional latent sphere. We identify two limitations of the original formulation that reduce its generation quality. First, random points concentrate near the equator relative to the pole on an encoded latent, but the training rotation never reaches this region, leaving a gap that limits one-step generation. Second, training for generation with pixel-wise reconstruction loss encourages the decoder to average over plausible images, producing blurry images that lack high-frequency details. We present Sphere Encoder 2 to address both limitations, substantially improving image generation quality while maintaining the speed and simplicity of a autoencoder. Models are released at \href{https://github.com/kaiyuyue/sphere2}{github.com/kaiyuyue/sphere2}.

1 INTRODUCTION

Sphere Encoder 2 addresses the original model’s equatorial training gap and blurry one-step outputs by changing its noise parameterization and generation objective. It achieves leading FDr6 results on ImageNet at 256 × 256 and 512 × 512 with substantially less sampling compute than pixel diffusion models.

1 INTRODUCTION

  • The original Sphere Encoder produces blurry single-step generations because its training noise stops at 85◦ and pixel-wise reconstruction loss averages over plausible images.Iterative encoder-decoder refinement sharpens images, but soft edges remain even after as many as four passes.Source: §1 · p. 2
  • Sphere Encoder 2 reaches 90◦ noise robustly and separates low-noise reconstruction from high-noise generation, dropping pixel loss and using latent score matching in the latter regime.The score-matching loss encourages outputs to lie on the image manifold without adversarial training.Source: §1 · p. 2
  • Sphere Encoder 2 produces sharp one-step images at 512 × 512 resolution on ImageNet.The paper presents this as the result of addressing the equatorial coverage gap and reconstruction-loss blur.Source: §1 · p. 2
  • Sphere Encoder 2 achieves the best FDr6 at 256 × 256 and the second best at 512 × 512 among pixel-space ImageNet generators, using 23× to 40× less sampling compute than pixel diffusion models.It uses only 4×2 network evaluations, compared with 200 for JiT and PixNerd.Source: §1 · p. 2

2 RELATED WORK

Related work spans autoencoders that need a learned prior or a second generation stage, one/few-step methods, and approaches to distribution matching. Sphere Encoder 2 builds on direct sphere-to-pixel generation and introduces a lightweight score-matching loss trained without a pretrained teacher.

2 RELATED WORK

  • Earlier autoencoders either fall short of diffusion and GAN quality on ImageNet, rely on a pretrained tokenizer, or use a second-stage prior; Sphere Encoder revisits single-stage generation but produces blurry images.Sphere Encoder 2 remains single-stage while removing the blur.Source: §2 · pp. 2–3
  • Prior spherical-latent approaches include small-scale von Mises–Fisher posteriors, learned priors over spherical latents, and direct sphere-to-pixel decoding limited to MNIST and CelebA; Sphere Encoder scales direct decoding to ImageNet.Sphere Latent Encoder is two-stage and uses a RAE decoder to reach pixels.Source: §2 · p. 3
  • One/few-step generators include distillation and from-scratch methods, while Sphere Encoder 2 is trained from scratch and adds steps by encoding, rotating, and decoding.One step decodes a random point on the sphere directly into pixels.Source: §2 · p. 3
  • Adversarial distribution matching is effective but needs careful tuning and can be unstable; DMD requires a pretrained diffusion teacher, while SMT operates in pixel space at 64 × 64 or below.Sphere Encoder 2 proposes a lightweight loss that trains real and fake score models online in frozen feature space without a pretrained teacher.Source: §2 · p. 3

3 PRELIMINARIES: SPHERE ENCODER

Sphere Encoder maps images to spherical latents and decodes them back; random sphere samples can be decoded to generate images. Its training noise leaves out the equatorial band used by random samples, creating a gap that limits one-step generation.

3 PRELIMINARIES: SPHERE ENCODER

  • Sphere Encoder is a non-variational autoencoder whose encoder maps an image to a latent that is projected onto a global sphere, and whose decoder maps the spherical latent back to an image.The latent has N tokens with feature dimension D, and the unit sphere is in R^d.Source: §3 · p. 4
  • Generation samples a Gaussian vector, spherifies it to obtain a uniform point on the sphere, and decodes that point into an image.This relies on the latent space being approximately uniformly distributed on the sphere.Source: §3 · p. 4

3.1 GAP TO THE EQUATOR

  • To bridge encoded training latents and continuous uniform sphere samples, Sphere Encoder adds noise to a clean latent before train-time reconstruction, sampling α uniformly from [0, 85◦].Here α approximates the angle between the clean latent and its noisy version in high dimensions.Source: §3.1 · p. 4
  • Random points on a high-dimensional sphere lie near the equator, but the 85◦ training limit leaves the final 5◦ band unseen and limits one-step generation.Setting αmax to 90◦ is not viable in the original formulation because tan(α) diverges as α approaches 90◦.Source: §3.1 · p. 4

4 SPHERE ENCODER 2

Sphere Encoder 2 closes the training gap to the latent sphere’s equator and separates reconstruction from generation, using distinct losses to preserve semantic consistency and improve fidelity. Its decoder remains part of a single encoder–decoder autoencoder, with one-step generation at the equator and iterative refinement for few-step generation.

4.1 ROTATION TO THE EQUATOR

  • Explicit rotation reaches 90°, training the decoder on the equatorial band and matching the distribution of one-step generation latents.At 0° the rotated latent is the clean encoded latent; at 90° it is the random direction F(e).Source: §4.1 · p. 4

4.2 STILL AN AUTOENCODER

  • A frozen DINOv3-small encoder and a trained ViT decoder retain the autoencoder design; few-step sampling repeatedly encodes, rotates toward a fixed noise direction, and decodes.The default sampling rotation angle is 85°; one-step generation decodes a random point on the sphere.Source: §4.2 · p. 5

4.3 ANGLE CUTOFF: TWO REGIMES ON ONE SPHERE

  • The cutoff separates low-angle reconstruction, which retains pixel-wise loss, from near-equator generation, which drops it and uses a smooth loss-weight transition.The sigmoid transition is centered at the cutoff; β sets its width, with the default cutoff at 70° and β at 5°.Source: §4.3 · pp. 5–6, Figure 4 · p. 6

4.4.1 SEMANTIC ALIGNMENT

  • Semantic alignment uses ImageNet-pretrained ConvNeXt logits to keep generated images consistent with their inputs without labels or additional loss-forward cost.The target is the input image itself, and the loss shares the LPIPS forward in the reconstruction regime.Source: §4.4.1 · p. 6
  • Generation quality measured by FID and FDr6 peaks at a 70° cutoff, while reconstruction quality degrades once the cutoff moves past 45°.The authors use a 70° cutoff by default; beyond 45°, sin(α) exceeds cos(α), and noise overwhelms the clean latent.Source: §4.4.1 · p. 6, Figure 5 · p. 7

4.4.2 FIDELITY IMPROVEMENT

  • Two lightweight score models estimate real and generated distributions in frozen image features; their difference guides decoder updates toward real features.The models each have two transformer layers, are frozen during transport, and are run once each; the decoder alone is updated through the score loss.Source: §4.4.2 · pp. 7–8, Figure 7 · p. 8

4.4.3 LATENT CONSISTENCY

  • Latent consistency is retained using the frozen DINOv3-small encoder, adding one inexpensive encoder forward on the generated image.This matches the generated image’s latent to the input latent and provides semantic alignment in latent space.Source: §4.4.3 · p. 8

4.4.4 TRAINING OBJECTIVE

  • The decoder objective combines semantic alignment, score matching, and weighted latent consistency alongside the reconstruction-regime loss.The stated objective includes Lsem + Lscore + 0.1 Llat-con; the reconstruction regime retains its pixel-wise loss.Source: §4.3 · p. 6, §4.4.4 · p. 8

5 EXPERIMENTS

Sphere Encoder 2 compares favorably with prior sphere encoders on Oxford Flowers and with pixel-space generators on ImageNet, including at low sampling compute. The results also show a gap between FDr6 and gFID, which the paper investigates with an additional loss.

Evaluation

  • The evaluation generates 50K images per dataset and reports FDr6 and gFID against the whole training set, prioritizing FDr6 as an average over six feature spaces.For Sphere2, sampling and guidance angles are chosen by FDr6, and gFID is reported at that same setting.Source: §5 · p. 9

Oxford Flowers

  • On Oxford Flowers 256 × 256, one-step no-CFG Sphere2-B outperforms both prior sphere encoders at four steps with CFG, using 6× fewer parameters than Sphere Encoder.Sphere2-L improves FDr6 over Sphere2-B at every step count but has worse gFID, so the two metrics do not always correlate.Source: Table 1 · p. 9, §5 · p. 9

ImageNet

  • Sphere2 leads on FDr6 but has higher gFID than other models; adding LFD-lite lowers gFID for every model and improves FDr6 in most settings.The authors attribute the metric gap to score matching in ConvNeXt V2-N features transferring to five modern FDr6 spaces but not the older Inception space.Source: §5 · p. 10
  • At 256 × 256, Sphere2-L at 4 NFE outperforms JiT-H and pMF-H and ranks second only to PixNerd; the same ordering holds at 512 × 512.The reported comparisons prioritize FDr6; Sphere2-L's 4-NFE result is 5.77, versus 7.66 for JiT-H, 6.87 for pMF-H, and 5.01 for PixNerd.Source: §5 · pp. 9–10, Table 2 · p. 10

Sampling compute and speed

  • Without CFG, Sphere2 improves FDr6 for equal or fewer GFLOPs; at 512 × 512 with CFG, it samples 13× faster than JiT-L and 66× faster than RAE-DiTDH-XL/2.The speed comparison is end-to-end sampling of 50K images at batch size 16.Source: Table 3 · p. 11, §5 · p. 11, Figure 9 · p. 11

6 CONCLUSION

Sphere Encoder 2 extends spherical autoencoders with an explicit rotation to the equator and separate reconstruction and generation regimes, using semantic alignment, latent consistency, and score matching for sharper outputs without adversarial training. The conclusion presents it as a complementary one/few-step route, while interpolation examples illustrate class transitions and refinement from non-ImageNet inputs.

6 CONCLUSION

  • The generation regime combines semantic alignment and latent consistency with lightweight score matching to produce semantically faithful but structurally different images without adversarial training.The conclusion describes the approach as splitting the rotation arc into reconstruction and generation regimes on the same sphere.Source: §6 · p. 11
  • Sphere Encoder 2 adds a complementary path by making the autoencoder a generator in its own right and could be viable for one/few-step generation.The paper situates this path alongside consistency models, MeanFlow, drifting models, and GANs.Source: §6 · p. 12

A INTERPOLATION

  • Class interpolation fixes a random point and slerps between two class embeddings; rows progress from similar class pairs to dramatic pairs.The interpolation includes five points between the two embeddings.Source: Figure A14 · p. 16, A INTERPOLATION · p. 16
  • For non-ImageNet inputs, refinement under a target ImageNet class moves outputs toward that class; at α = 85°, outputs are sharper and drift toward the ImageNet manifold.At α = 0°, the decoder reconstructs the input from its clean latent; each subsequent refinement step moves the output closer to the target class.Source: Figure A15 · p. 17, A INTERPOLATION · p. 17

B.1 DISTRIBUTION SHIFT

LFD-lite regularizes feature statistics to address drift concentrated in Inception space, while its effect varies by model size and the decoder's angle conditioning improves metrics. The section also distinguishes this regularizer from the original FD loss and reports that the visual difference can be imperceptible.

B.1 DISTRIBUTION SHIFT

  • Inception-space FDr drops 34–46% with LFD-lite for both models at every step count, while the other five modern spaces change mostly by single digits.This Inception-specific shift corresponds to the gFID improvement; the supplied table reports FDr6 by feature space and sampling step.Source: §B.1 · p. 18, Table A4 · p. 18
  • Sphere2-B benefits in every space and step count, but Sphere2-L's overall FDr6 worsens 17% at one step despite improved Inception alignment.For Sphere2-L, ConvNeXt, MAE, SigLIP, and CLIP all degrade at one step; multi-step sampling recovers the loss as each refinement re-encodes the image.Source: §B.1 · pp. 18–19
  • LFD-lite makes Sphere2-B's generations visually indistinguishable from those without the loss, despite aligning feature statistics with real data in Inception space.The loss is aimed at gFID's feature space; the authors say it can be dropped when evaluation emphasizes human preference or prompt following instead.Source: §B.1 · p. 19
  • LFD-lite regularizes feature statistics during training; it uses a small weight of 0.1 alongside three other losses rather than serving as the main post-training objective.It reuses two extractors already used by other losses and adds one Inception forward with almost no compute.Source: §B.1 · pp. 18–19

B.2 ANGLE CONDITIONING

  • Angle conditioning improves metrics because it lets the decoder distinguish latent angles, countering its hypothesized tendency to hedge toward the mean.The ablation in Table A5 supports the authors' hypothesis, and they retain angle conditioning by default.Source: §B.2 · p. 22

B.3 ANGLE DISTRIBUTION IN TRAINING

The training angle density is controlled by a single parameter, but concentrating samples toward the equator does not improve the overall results. Uniform angle sampling is the simplest and most effective choice in the reported Oxford Flowers experiments.

B.3 ANGLE DISTRIBUTION IN TRAINING

  • The parameter k sets the training-angle density: positive k concentrates samples near the equator, while negative k concentrates them near the pole.k = 0 corresponds to uniform sampling.Source: §B.3 · p. 22
  • Uniform angle sampling (k = 0) gives the best 1-step gFID and the best FDr6 at every step count.The figure compares generation quality across k values and identifies uniform sampling as best.Source: §B.3 · p. 22, Figure A20 · p. 22
  • Concentrating samples near the equator buys only a marginal multi-step gFID gain at k = 5; the authors attribute the weak need for reweighting to angle-specific losses for each regime.They therefore use uniform sampling by default.Source: §B.3 · p. 22

B.4 SAMPLING ANGLE AT INFERENCE

Inference angle controls one-step diversity, while the reported multi-step angle choices prioritize FDr6 when it disagrees with FID. The section also describes spherical guidance and identifies the encoder and score-feature configurations used for quality and efficiency.

B.4 SAMPLING ANGLE AT INFERENCE

  • One-step sampling angle controls diversity: at 90°, outputs nearly coincide, while angles below 90° retain reference information and produce more diverse images.Across the sweep, FDr6 and FID agree on a U-shaped quality trend, with best quality at 86°.Source: §B.4 · p. 23, Figure A22 · p. 23
  • For multi-step generation, Sphere2-B uses α = 85.5° and Sphere2-L uses α = 84° at both resolutions to prioritize FDr6 when FDr6 and FID disagree.The cited table and figure describe the guidance-angle sweep and the sampling angles used for the reported results.Source: §B.4 · p. 23, Figure A23 · p. 24, Table A6 · p. 24

B.5 SPHERICAL CFG

  • Spherical CFG rotates the conditional latent away from the unconditional latent along their great circle, keeping the guided latent on the sphere.Guidance angle γ fixes the rotation step; γ = 0 recovers the unguided procedure, and guidance requires at least two sampling steps.Source: §B.5 · pp. 23–24

B.6 FROZEN LATENT SPACE

  • DINOv3 outperforms SigLIP2 at every tested size, and DINOv3-small at its original dimension of 384 gives the best quality-efficiency trade-off.The encoder is run at every sampling step; Table A7 compares pretrained encoders and latent dimensions.Source: §B.6 · p. 24, Table A7 · p. 26

B.7 SCORE FEATURE SPACE

  • For Lscore, combining a middle and a deep feature layer gives the best capacity trade-off for both ConvNeXt V2-N and DINO-S/8.Adding more layers costs up to twice the parameters and helps only a little; Table A8 reports feature extractors and layer selections.Source: §B.7 · p. 26, Table A8 · p. 26

B.8 SCORE SIGMA DISTRIBUTION

The score-loss transport uses noise levels chosen to remain within the denoisers' training range while emphasizing small-noise detail. The implementation standardizes frozen features, computes a per-sample transport direction, and balances layer contributions.

B.8 SCORE SIGMA DISTRIBUTION

  • The best transport noise distribution is q(σ) = (−1.5, 1.0), shifted toward smaller σ and slightly narrower than training p(σ); shifting further worsens FDr6.The transport distribution is chosen to stay within the bulk of p(σ) while leaning toward noise levels where score differences carry fine detail.Source: §B.8 · p. 26, Figure A25 · p. 26, Table A9 · p. 27

C.1 DIFFERENTIABLE AUGMENTATION

  • Differentiable horizontal flips and translations augment decoder outputs before frozen-feature extraction; translation smooths patch-boundary effects for DINO-S/8.The translation ratio is 0.015625, corresponding to up to 4 pixels at 256 × 256 and 8 pixels at 512 × 512.Source: §C.1 · p. 27

C.2 FEATURE STANDARDIZATION

  • Each scored feature layer is standardized using statistics estimated on real features, then applied to both real and generated features.The running statistics are accumulated for 1000 steps and frozen; the standardized features have zero mean and standard deviation σdata.Source: §C.2 · pp. 27–28

C.3 TRANSPORT DIRECTION

  • The transport direction is computed per sample and normalized by the real score model's mean absolute denoising error over feature entries, with a floor guarding against near-zero error.The noise level is drawn from q(σ) and clamped to [0.002, 8.0]; both score models are frozen when computing the direction.Source: §C.3 · p. 28

C.4 TRANSPORT BALANCE

  • Layer balancing tracks each layer's gradient scale and sets weights so that w_j a_j = T, equalizing layer contributions while holding the overall gradient magnitude at its calibrated level.The target is estimated during an initial training period and then frozen; weights are clipped to [0.05, 20] to guard against a collapsed layer receiving unbounded weight.Source: §C.4 · p. 28

D IMPLEMENTATION DETAILS

The appendix records the architecture, training, and sampling configurations used for the Oxford Flowers and ImageNet experiments, along with the representation models used to compute FDr6 at two resolutions.

D IMPLEMENTATION DETAILS

  • Configurations for Sphere2-B and Sphere2-L cover the Oxford Flowers and ImageNet results reported in Tables 1 and 2.Table A10 indicates that values in merged cells are shared by both models.Source: D IMPLEMENTATION DETAILS · p. 29, Table A10 · p. 29
  • Six representation models are used to compute FDr6 at 256 × 256 and 512 × 512 on Oxford Flowers and ImageNet.The 256 × 256 model set follows the FDr6 setup of Yang et al. (2026), originally for ImageNet 256 × 256.Source: D IMPLEMENTATION DETAILS · p. 29, Table A11 · p. 30
  • For FDr6 at 512 × 512, each representation-model family uses its highest-resolution checkpoint; Inception-v3 and MAE lack such checkpoints and retain those from Table A11.Source: Table A12 · p. 30

Related papers

Loading 2610.02208v1…