PaperlayerView exact revision on arXiv

Source-linked AI summary

SphereVAE: Hyperspherical Latent Autoencoders for Robust Autoregressive Speech Representation Modeling

Haoyu Zhang, Jingbin Hu, Hanke Xie, Qirui Zhan, Wenhao Li, Ziyu Zhang, Xiaming Ren, Yue Li, Xunyu Zhu, Zhipeng Chen, Lei Xie

arXiv:2609.09903v1eess.AS

TL;DR

Discrete codecs stabilize prediction but can lose fine-grained acoustic information, while continuous latent prediction can incur vector-space shifts. SphereVAE constrains VAE latents to a unit hypersphere and, in VoxCPM evaluations, reports lower content error rates with comparable speaker similarity and stronger long-text retention.

  • Problem

    Discrete codecs can discard fine-grained acoustic detail, while continuous latent prediction can incur vector-space shifts.

    Source: §1 · p. 2

  • Method

    SphereVAE models continuous speech latents with a Power Spherical posterior on the unit hypersphere and regularizes it toward a uniform prior.

    Source: §1 · p. 3, §3.1 · p. 6

  • Results

    5.305% WER on EN and 1.141% CER on ZH are the lowest content error rates among the compared VAE representations, with comparable speaker similarity and stronger long-text retention.

    Source: §4.3 · p. 9, §5 · p. 13

  • Takeaways & Limitations

    Latent-space geometry is a meaningful design axis for continuous speech representations.

    Source: §5 · p. 13

    The hyperspherical constraint reduces reconstruction freedom, and SphereVAE underperforms the standard VAE on reconstruction-oriented metrics.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

With the rapid development of speech generation technology, discrete codec representations have been widely used because they provide a stable prediction paradigm. In expressive speech generation, however, the quantization bottleneck of discrete codecs results in information gaps in fine-grained prosody, timbre, pronunciation, and frame-to-frame continuity. Continuous representations (e.g., VAE latents), by eliminating this constraint, have emerged as a more effective alternative for autoregressive modeling. Yet when continuous representations are used as autoregressive prediction targets, prediction errors can accumulate along the generation chain, causing latent drift and degrading long-form stability. To mitigate this problem, we propose SphereVAE, which constrains the VAE latent space to the unit hypersphere. SphereVAE defines a Power Spherical posterior on the hypersphere and regularizes the latent distribution toward a uniform prior, so that information is encoded mainly by directional variation, providing a bounded geometric target for autoregressive prediction and reducing the risk of norm drift. SphereVAE underperforms the standard VAE on reconstruction metrics due to reduced latent freedom. However, when integrated into VoxCPM for zero-shot TTS and long-text generation, it yields lower content error rates with comparable speaker similarity, and shows more stable long-range speaker consistency. These results indicate that an appropriate latent geometric constraint can effectively mitigate autoregressive error accumulation and drift in speech generation.

1 Introduction

Speech representations shape both reconstruction and downstream sequence modeling: discrete codecs stabilize prediction but can discard fine-grained acoustic detail, while continuous latents face autoregressive drift. SphereVAE constrains continuous latents to a unit sphere, with reported gains in content prediction and long-text speaker-similarity retention.

1 Introduction

  • Discrete codec tokens provide bounded prediction but may lose fine-grained prosody, timbre, pronunciation, and frame-to-frame smoothness.Finite codebooks stabilize generation by limiting each step to codebook entries or a small set of discrete units.Source: §1 · p. 2
  • Continuous latents preserve a more flexible acoustic space, but autoregressive prediction errors can shift representations and propagate through later steps.Each previous prediction conditions the next, making long-form speech generation sensitive to error accumulation.Source: §1 · pp. 2–3
  • SphereVAE achieves the lowest content error rates among the compared VAE variants in zero-shot TTS and better retains speaker similarity during long-text inference.The introduction reports these results while positioning the method as a constrained continuous latent space for autoregressive generation.Source: §1 · p. 3
  • SphereVAE fixes latent norm and organizes speech information mainly through directional variation, providing a bounded target for autoregressive prediction.The paper frames this geometric constraint as a way to reduce latent drift while preserving continuous representations.Source: §1 · p. 3

2 Related Work

Prior work develops continuous speech tokenizers and autoregressive generation systems, but long-horizon error accumulation in continuous representations remains a challenge. SphereVAE addresses the representation geometry rather than the generator.

2.1 Continuous Speech Representation Learning

  • Continuous tokenizers avoid hard quantization and offer richer acoustic capacity, while learned representations also serve as interfaces that determine what downstream generators can use.Discrete codebooks can make fine-grained prosody, timbre, pronunciation, and temporal variation difficult to preserve.Source: §2.1 · p. 4
  • Standard VAE, SemanticVAE, and σ-VAE approaches improve continuous tokenization through reconstruction, semantic alignment, or fixed latent variance, but do not explicitly address autoregressive error accumulation.SphereVAE draws on hyperspherical VAE and Power Spherical work to change latent geometry, organizing information through direction while fixing norm.Source: §2.1 · p. 4

2.2 Autoregressive Modeling of Continuous Representations

  • Continuous autoregressive TTS work includes patch-based diffusion prediction, hybrid acoustic-semantic tokenization, hierarchical generation, and self-correction techniques.DiTAR, VibeVoice, VoxCPM, and Dots.tts represent different approaches to continuous or hybrid generation.Source: §2.2 · pp. 4–5
  • Long-horizon prediction errors remain a central difficulty; SphereVAE focuses on whether changing the representation prior can reduce drift and improve long-form robustness.This focus is described as orthogonal to work on generators, alignment, semantic conditioning, and local diffusion prediction.Source: §2.2 · p. 5

3 Method

SphereVAE replaces a Euclidean Gaussian bottleneck with a unit-hypersphere latent modeled by a Power Spherical posterior. Its training objective combines speech reconstruction with hyperspherical regularization and adversarial supervision.

3.1 Hyperspherical Latent Space Formulation

  • SphereVAE encodes speech into direction and concentration parameters, normalizes the direction to the unit hypersphere, and samples a Power Spherical latent for decoder reconstruction.The concentration parameter controls how tightly samples cluster around the mean direction.Source: §3.1 · pp. 5–6
  • The posterior is regularized toward a uniform hyperspherical prior, while fixed norm forces speech information to rely primarily on directional variation.This trades some reconstruction freedom for a clearer geometric boundary that limits growth through the norm dimension during autoregressive prediction.Source: §3.1 · p. 6

3.2 Training Objective

  • Training combines waveform reconstruction, hyperspherical KL regularization, generator adversarial loss, and feature matching.The encoder samples from the Power Spherical posterior before the decoder reconstructs the input waveform.Source: §3.2 · pp. 6–7
  • Multi-scale mel-spectrogram L1 loss compares input and reconstructed speech, capturing global spectral envelope and local time-frequency detail.It serves as the primary reconstruction signal.Source: §3.2 · p. 7
  • The multi-scale STFT adversarial term penalizes perceptual artifacts beyond pointwise spectral distances, while feature matching compares discriminator representations of real and reconstructed speech.The discriminator feature term also stabilizes GAN training and encourages the decoder to reproduce multiresolution acoustic characteristics.Source: §3.2 · p. 7

4 Experiments

The experiments evaluate SphereVAE’s reconstruction trade-off, zero-shot TTS performance, long-text speaker retention, and latent geometry. SphereVAE has lower content error rates and stronger long-range speaker retention, while its hyperspherical constraint reduces reconstruction freedom.

4.1 Experimental Setup

4.2 VAE Reconstruction Results

  • VAE leads reconstruction-oriented metrics, while SemanticVAE has the highest SIM; SphereVAE remains usable but trades reconstruction freedom for constrained geometry.The reconstruction results are on LibriSpeech-PC test-clean.Source: §4.2 · p. 9, Table 1 · p. 9

4.3 Zero-shot TTS Model Evaluation

  • SphereVAE records 5.305% WER on EN and 1.141% CER on ZH, the lowest content error rates among the four VAE representations compared.VAE is the closest EN WER baseline at 5.591%, while SemanticVAE is the closest ZH CER baseline at 1.241%; SIM differences are small.Source: §4.3 · p. 9, Table 2 · p. 10

4.4 Long-Text Inference Evaluation

  • SphereVAE achieves the highest M_SIM and SIMlate over 0–27 s, preserving higher speaker similarity with a smaller decline than standard VAE and SemanticVAE.The results cover the first nine 3-second segments; later segments are excluded because valid-pair counts drop sharply for some models.Source: Table 3 · p. 10, §4.4 · pp. 10–11

4.5 Latent Distribution Analysis with t-SNE and PCA

  • SphereVAE shows stronger speaker clustering and higher cumulative explained variance than VAE, with more compact within-speaker structure and clearer between-speaker separation.Quantitatively linking latent geometry to autoregressive drift metrics would further strengthen this interpretation.Source: §4.5 · pp. 11–12, Figure 3 · p. 12

5 Conclusion

SphereVAE constrains continuous speech latents to the unit hypersphere, fixing their norm and providing a bounded target for autoregressive prediction. The conclusion reports improved downstream generation behavior and stronger speaker-similarity retention in long-text inference, while noting that reconstruction quality is not its objective.

5 Conclusion

  • SphereVAE fixes latent norm and organizes information through directional variation, giving autoregressive prediction a bounded geometric target.The method investigates error accumulation in autoregressive modeling of continuous speech representations.Source: §5 · pp. 12–13
  • SphereVAE achieves the lowest SeedTTS-eval content error rates among VAE, SemanticVAE, and σ-VAE, with comparable speaker similarity and stronger retention in long-text inference.The paper positions the method as improving downstream generation behavior rather than maximizing one-step reconstruction quality.Source: §5 · p. 13
  • PCA and t-SNE analyses suggest that the hyperspherical constraint concentrates representational variation into more dominant directions and changes latent-space organization.Fig. 4 compares cumulative explained variance between SphereVAE and standard VAE latent features.Source: §5 · p. 13, Figure 4 · p. 13
  • Future work will develop quantitative drift metrics and evaluate SphereVAE under longer and more diverse generation conditions.Source: §5 · p. 13
Loading 2609.09903v1…