Learning Latent Representations for Speech Generation and Transformation

1. Introduction

• Speech varies with linguistic content, speaking style, dialect, speaker identity, emotion, environment, and channel, motivating automatic factor learning.Understanding these influences supports adaptation, data augmentation, voice conversion, and speech compression.Source: §1 · p. 1
• The paper applies a convolutional VAE to learn latent representations through an unsupervised probabilistic model of speech generation.The approach targets a research area with less prior work than deep generative modeling for natural images.Source: §1 · p. 1
• Latent-space arithmetic is used to decompose representations into attributes such as speaker identity and linguistic content.The operations manipulate one attribute while aiming to keep remaining attributes fixed.Source: §1 · p. 1
• Experiments measure whether speaker characteristics can change without changing linguistic content, and vice versa.The paper also analyzes generation of speech segments with different durations.Source: §1 · p. 1

2. Related Work

• WaveNet and SampleRNN generate audio by predicting each next sample from preceding samples rather than learning latent representations for entire segments.WaveNet uses dilated causal convolutions, while SampleRNN uses multi-scale recurrent layers.Source: §2 · p. 1
• Earlier VAE speech studies extracted frame- or utterance-level features or learned frame-level latent representations for robust recognition and smooth sample generation.The paper positions its work within comparatively limited prior speech research using VAEs.Source: §2 · p. 1

3.1. Variational Autoencoder

• The VAE introduces qφ(z|x) to approximate the intractable posterior pθ(z|x), optimizing the variational lower bound.The bound combines a KL-divergence regularizer with expected reconstruction log likelihood.Source: §3.1 · pp. 1–2
• The model assumes diagonal Gaussian recognition and generative distributions with a centered isotropic Gaussian prior pθ(z) = N(z; 0, I).The means and covariances are computed by neural networks, while the prior has no free parameters.Source: §3.1 · p. 2

3.2. Proposed Model Architecture

• Speech segments are fixed-length frame sequences whose latent variable is intended to encode spoken content, speaker identity, and channel effects.This defines the representation-learning target for the proposed architecture.Source: §3.2 · p. 2
• The recognition network treats each segment as a time-by-frequency image, using frequency-spanning first-layer filters and time-axis down-sampling.The generative network mirrors the recognition network and predicts Gaussian parameters for the observed data.Source: §3.2 · p. 2
• The architecture uses tanh in non-Gaussian layers and batch normalization outside Gaussian parameter layers to avoid overflow in the KL divergence and conditional likelihood.Gaussian parameter layers remain unactivated because their means and log-variances are unbounded.Source: §3.2 · p. 2

4.1. Deriving Latent Attribute Representations

• Latent attribute representations are estimated from posterior samples of speech segments sharing an attribute value, such as the phone /ae/.The representation μr is defined from the conditioned latent prior and estimated by averaging sampled latent representations.Source: §4.1 · p. 2
• Averaging latent samples is argued to yield a reasonable speech-like latent point when the sampled points have sufficiently high prior probability.The argument uses the log-concavity of the prior.Source: §4.1 · p. 2

4.2. Arithmetic Operations to Modify Speech Attributes

• The method assumes each speech attribute occupies an independent orthogonal latent subspace, allowing the full representation to decompose into attribute-specific components.This assumption is the basis for deriving arithmetic operations that modify selected attributes.Source: §4.2 · p. 2
• To change an attribute from rs to rt, the method adds the shift vrs→rt = μrt − μrs to the encoded latent representation.The construction is intended not to modify latent representations of other attributes.Source: §4.2 · p. 2

5.1. TIMIT

• TIMIT provides 6,300 utterances from 630 speakers, split into 462/50/24 non-overlapping train, development, and test speakers.The corpus contains 5.4 hours of broadband read speech, with approximately 70% male and 30% female speakers and 58 phones after silence-phone exclusion.Source: §5.1 · p. 3

5.2. Data Preprocessing

• The study uses 25ms Hanning windows with 10ms shifts to compute magnitude spectra or 80-bin Mel filter-bank features.Spectral coefficients are floored at -20dB, while the filter banks preserve more detail in lower-frequency regions.Source: §5.2 · p. 3
• Speech segments span 200ms or 1s, corresponding to 20-frame syllable-level and 100-frame word-level datasets.Source: §5.2 · p. 3

6.2. Latent Attribute Representation

• VAE reconstructions preserve more harmonic structure and clearer spectral envelopes than AE and Fbank averages, whereas AE reconstructions show frequent vertical stripe artifacts.Source: §6.2 · p. 3, §6.1 · p. 3, Figure 1 · p. 3

6.3. Modifying Attributes of Speech

• Speaker and phone representations occupy low-cosine-similarity off-diagonal blocks, while phone representations cluster by phonetic characteristics.These patterns support approximately orthogonal speaker and phone subspaces and suggest further structure within the phone subspace.Source: §6.3 · p. 3, Figure 2 · p. 3
• Latent shifts modify phonetic content or speaker identity while retaining the other attribute: /aa/ to /ae/ raises F2, and female-to-male conversion preserves the spectrum envelope.The /aa/-to-/ae/ transformation keeps harmonics roughly stable, while the female-to-male transformation reduces harmonics without changing phonetic content.Source: §6.3 · pp. 3–4, Figure 3 · p. 3, Figure 4 · p. 4
• After phone modification, the average /ae/ posterior reaches 30% while the average correct speaker posterior degrades only slightly; speaker modification shifts the target posterior to 29% madc0.Table 2 reports these average posterior changes over modified speech segments, with the speaker and phone transformations affecting their intended attributes more than the fixed ones.Source: §6.3 · p. 4, Table 2 · p. 4

6.4. Random Sampling from the Latent Space

• Syllable-level random samples look reasonable, but word-level samples are less natural and exhibit excessive closures; the word-level latent prior also has higher cross-dimension correlations.The covariance comparison suggests that the assumed diagonal Gaussian prior is less appropriate for the word-level model.Source: §6.4 · p. 4, Figure 5 · p. 4, Figure 6 · p. 4

6.5. Walking in the Latent Space

• The evaluation also includes latent-space interpolation with VAE and AE using 200ms speech segments.Source: Figure 7 · p. 4

• A convolutional VAE models speech generation and learns latent representations in an unsupervised framework.Source: §7 · p. 4
• The learned representations can be decomposed and speech attributes can be modified, with both capabilities demonstrated qualitatively and quantitatively.Source: §7 · p. 4
• Future work will extend the approach with hierarchical recurrent models to capture information at different time scales and generate variable-length speech.Source: §7 · p. 4