Source-linked AI summary
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
Wei-Ning Hsu, Yu Zhang, James Glass
TL;DR
The paper asks how to learn disentangled and interpretable representations from sequential data without supervision. It uses a factorized hierarchical variational autoencoder with different priors for sequence-level and segment-level latent variables, achieving speaker-verification gains and up to 35% word-error-rate reduction in mismatched speech-recognition settings.
Problem
Learning disentangled and interpretable representations from sequential data without supervision remains insufficiently explored.
Method
A factorized hierarchical variational autoencoder assigns sequence-level and segment-level attributes to separate latent variables using sequence-dependent and sequence-independent priors.
Results
The model outperforms an i-vector baseline for speaker verification and reduces word error rate by up to 35% in mismatched speech-recognition scenarios.
Takeaways & Limitations
The learned representations support qualitative transformations of speakers or linguistic content through manipulation of different latent-variable sets.
Takeaways & Limitations
The quantitative evaluation uses two speech corpora, and TIMIT is not a standard corpus for speaker verification.
Abstract
from arXiv · showhide
We present a factorized hierarchical variational autoencoder, which learns disentangled and interpretable representations from sequential data without supervision. Specifically, we exploit the multi-scale nature of information in sequential data by formulating it explicitly within a factorized hierarchical graphical model that imposes sequence-dependent priors and sequence-independent priors to different sets of latent variables. The model is evaluated on two speech corpora to demonstrate, qualitatively, its ability to transform speakers or linguistic content by manipulating different sets of latent variables; and quantitatively, its ability to outperform an i-vector baseline for speaker verification and reduce the word error rate by as much as 35% in mismatched train/test scenarios for automatic speech recognition tasks.
1 Introduction
The paper targets disentangled, interpretable representations for sequential data without supervision by introducing a factorized hierarchical variational autoencoder. Evaluations on speech corpora show latent-variable factorization and improvements in speaker verification and mismatched speech recognition.
- The paper addresses the limited research on learning disentangled representations from sequential data without supervision.
- The proposed factorized hierarchical variational autoencoder explicitly models multi-scaled information with a factorized hierarchical graphical model.
- The inference model optimizes at the segment level rather than the full sequence level, improving scalability for very long sequences.
- The model factorizes sequence-level and segment-level attributes into different latent-variable sets, enabling transformations by manipulating those variables.
- 2.38% and 1.34% equal error rates were achieved on unsupervised and supervised speaker verification, outperforming an i-vector baseline.
- Up to 35% word-error-rate reduction was obtained in mismatched train/test scenarios for speech recognition.
2 Factorized Hierarchical Variational Autoencoder
The FHVAE models sequence-level and segment-level attributes with separate latent variables and priors, enabling scalable unsupervised disentanglement in sequential data.
- Multi-scale factorization: Speech attributes operate at multiple temporal scales: speaker identity affects sequence-level F0 and volume, while phonetic content varies at the segment level.
- Latent-variable design: FHVAE assigns sequence-level attributes to latent sequence variables and segment-level attributes to latent segment variables using sequence-dependent and sequence-independent priors.
- Generative model: The generative process draws a sequence vector, samples latent sequence variables conditionally, samples segment variables independently, and generates observations from both latent sets.
- Latent-variable design: The factorized priors encourage latent sequence variables to remain close within a sequence while latent segment variables encode residual attributes across sequences.
- Optimization: Segment-level optimization improves scalability for extremely long sequences by avoiding computationally expensive sequence-level batched updates.
- Extensions: The two-scale hierarchy can be extended with additional variables for session-dependent or dataset-dependent priors.
- Discriminative objective: The model adds a discriminative objective because trivial sequence vectors would otherwise prevent z1 and z2 from separating sequence and segment attributes.
3 Sequence-to-Sequence Autoencoder Model Architecture
The Seq2Seq-FHVAE uses recurrent encoders and a recurrent decoder to stochastically represent and reconstruct subsequences while modeling temporal relationships.
- Architecture: The architecture uses recurrent encoders to capture within-subsequence temporal relationships and produce fixed-dimensional summaries.
- Architecture: A recurrent decoder generates frames step by step from the latent variables z1 and z2, forming a stochastic sequence-to-sequence autoencoder.
- Architecture: Figure 3 separates z1 and z2 encoders from the x decoder, using recurrent networks alongside fully connected layers for distribution parameters.
- Implementation: The formulation uses LSTMs for recurrent components and multilayer perceptrons to predict Gaussian means and variances.
4 Experiments
Experiments on TIMIT and Aurora-4 evaluate FHVAE's disentanglement qualitatively and quantitatively, covering latent-variable manipulation, speaker verification, and domain-mismatched speech recognition.
- Experimental setup: FHVAE experiments use speech corpora containing multiple information scales, including speaker, channel, noise, and linguistic content.The evaluation uses TIMIT and Aurora-4, with Aurora-4 spanning clean, channel, noisy, and combined conditions.
- Qualitative evaluation: Replacing a male utterance's sequence variable with a female utterance's sequence variable produces voice conversion while preserving linguistic content.The decoding experiment is conducted on Aurora-4 with α = 0.
- Qualitative evaluation: Varying latent segment and sequence variables preserves phonetic content across rows while changing speaker identity, and preserves speaker identity across columns while changing linguistic content.The experiments combine four segment variables with ten sequence variables to expose the factorization.
- Speaker verification: 16-dimensional µ2 s-vectors outperform i-vector baselines for speaker verification in both Raw and LDA settings, while 32-dimensional µ2 reaches 1.34% EER.Negative results from µ1 further support separation between utterance-level and segment-level attributes.
- Domain-invariant ASR: 6.6% lower PER on unseen female speakers is obtained with z1 than with FBank features when acoustic models are trained only on male speakers.FBank PER on the unseen domain increases by 16.1%, whereas z1 slightly degrades performance on male speakers but improves transfer to female speakers.
- Domain-invariant ASR: 30% lower WER is achieved on combined noise-and-channel data when z1 is used, despite training the FHVAE without that combined domain.Across mismatched Aurora-4 domains, z1 reduces WER by 16% to 35% relative to baseline, with less than 2% degradation on matched domains.
5 Related Work
Prior work extends VAEs to sequential and structured data, but whether their latent variables disentangle independent attributes remains unclear. β-VAE encourages disentanglement, yet does not provide interpretability without supervision.
- Structured VAEs: Dynamic Bayesian networks and hierarchical VAEs have modeled structured data, improving log-likelihood or generating reasonable samples.Examples include SRNN, VRNN, neural statistician, and SVAE.
- Open question: These prior models leave the disentanglement of independent attributes in latent space unresolved.
- Disentangled representations: β-VAE uses a boosted KL-divergence penalty to encourage disentanglement of independent attributes.
- Disentangled representations: β-VAE does not provide interpretability without supervision, motivating methods that learn both properties unsupervised.
6 Conclusions and Future Work
The paper introduces FHVAE for unsupervised disentanglement of sequence-level and segment-level attributes and verifies it on two speech corpora. Its derivation analytically computes Gaussian KL terms and rewrites the sequence variational lower bound, while future work extends hierarchy, training, and domains.
- Conclusions: FHVAE learns disentangled and interpretable representations for sequence-level and segment-level attributes without supervision.
- Conclusions: The model’s disentangling ability is verified qualitatively and quantitatively on two speech corpora.
- Future work: Future work includes deeper hierarchies, adversarial training for disentanglement, and applications to text and videos.
- Variational derivation: The expected KL-divergence between Gaussian distributions is computed analytically for the variational lower-bound derivation.
- Variational derivation: Replacing the analytically computed KL terms rewrites the variational lower bound for a sequence X.
B. Derivation of the Inferred S-Vector
For an unseen utterance, the sequence-level s-vector µ2 is inferred by approximating maximum a posteriori estimation with a conditional segment variational lower bound. The resulting objective is optimized through a concave quadratic with one maximum point.
- Inference: Inference of the sequence-level s-vector µ2 for an unseen utterance is formulated as an approximated maximum a posteriori estimation problem.
- Inference: The conditional segment variational lower bound L(θ, φ; ˜x(n)|µ2) approximates the conditional likelihood of each segment.
- Optimization: The optimal µ∗2 is derived from a concave quadratic function f(·) with exactly one maximum point.
C. FHVAE Model and Training Configurations
The Seq2Seq-FHVAE configuration uses compact recurrent and feedforward networks and trains with stochastic optimization. Training minimizes a discriminative segment variational lower bound with L2 regularization under specified optimizer settings.
- Architecture: Each Seq2Seq-FHVAE LSTM has one layer with 256 hidden units.Each MLP has one layer whose output dimension matches the variable whose mean or log variance it parameterizes.
- Latent configuration: The models use z2 variance σ²z2 = 0.25 and are tested with various dimensions for latent variables z1 and z2.
- Optimization: Training uses mini-batches of 256 to minimize the negative discriminative segment variational lower bound plus L2 regularization weighted 10^-4.
- Optimization: Adam training uses β1 = 0.95, β2 = 0.999, ϵ = 10^-8, and an initial learning rate of 10^-3.
D. Comparison of Seq2Seq-FHVAE and Alternative Architectures
The comparison evaluates alternative architectures for FHVAE reconstruction and examines latent-variable transformations. Recurrent architectures preserve speech detail better than the feed-forward alternative, while latent sequence variables support sequence-level transformations.
- Architecture comparison: Recurrent models outperform the feed-forward model on the TIMIT test set using fewer parameters.The comparison includes RNN, GRU, and LSTM architectures against a fully connected feed-forward network.
- Architecture comparison: LSTM reconstructions preserve sharper speech detail and superior high-frequency harmonic structure compared with FC reconstructions.The difference is visible in the reconstructed spectrograms, especially in the highlighted harmonic regions.
- Latent transformations: FHVAE examples vary latent variables to examine how segment-level and sequence-level information is represented.The supplied figure descriptions include experiments varying latent variables on TIMIT and Aurora-4.
- Latent transformations: Replacing a target sequence’s latent sequence-variable distribution with a reference sequence’s distribution enables speaker or noise-condition transformation.The proposed shift operates on Gaussian latent sequence distributions centered at their respective s-vectors.
F. More Details about the Speaker Verification Experiments
The speaker-verification evaluation uses equal error rate and compares the proposed system with an i-vector baseline extracted from MFCC-based features after voice activity detection.
- Evaluation metric: Equal error rate measures speaker-verification performance when false rejection and false acceptance rates are equal.The metric is defined directly for the verification evaluation.
- Baseline system: The baseline uses Kaldi i-vectors extracted from MFCCs with delta and delta-delta features after voice activity detection.The baseline also uses a full-covariance, gender-independent UBM with 2048 mixtures.
- Baseline system: The i-vector dimensionality is tuned on the development set for the baseline speaker-verification system.The passage describes this tuning as part of the baseline configuration.
G. More Details about the Domain Invariant ASR Experiments
The experiments use standard Kaldi GMM-HMM alignments and LSTM acoustic models, while qualitative FHVAE analyses show separate latent controls for segment- and sequence-level speech attributes.
- ASR experiments: GMM-HMM systems generate senone alignments for subsequent neural acoustic-model training in both TIMIT and Aurora-4.The neural acoustic model replaces the GMM acoustic model in the hybrid DNN-HMM system.
- Domain transformation: Replacing a noisy utterance’s latent sequence variables with those from a clean utterance produces denoised speech while preserving linguistic content.The result is demonstrated for Aurora-4 utterances with artificially added noise.
- Model configuration: The Seq2Seq-FHVAE configuration uses 32 dimensions for both z1 and z2 with α = 10 for discriminative training.The VAE comparison uses a 64-dimensional latent variable and a 512-unit LSTM encoder.
- Latent space traversal: Traversing latent segment variables changes segment-level attributes such as F2 movement, vowel type, and closure.The traversal experiments use five seed segments from the TIMIT test set with the remaining latent variables fixed.
- Latent space traversal: Traversing latent sequence variables changes harmonic patterns, volume, and formant-frequency offsets while largely preserving spectral contour and formant timing.These variables therefore capture interpretable sequence-level attributes distinct from the segment-level controls.