PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Speech Decomposition via Triple Information Bottleneck

Kaizhi Qian, Yang Zhang, Shiyu Chang, David Cox, Mark Hasegawa-Johnson

arXiv:2004.11284v6eess.AScs.LGcs.SD

TL;DR

Existing systems can separate timbre but leave content, pitch, and rhythm entangled, while obtaining labels for these components is difficult. SpeechSplit uses three carefully designed information bottlenecks to blindly decompose speech into four components and support aspect-specific style transfer without text labels. The paper reports successful disentanglement and conversion of intended aspects without altering the others, unlike AUTOVC’s timbre-only conversion.

  • Problem

    Existing voice conversion systems disentangle timbre but leave content, pitch, and rhythm mixed, while annotations for these components are difficult or expensive to obtain.

  • Method

    SpeechSplit uses an encoder-decoder with three channels, combining dimensional constraints and random time resampling as information bottlenecks.

  • Results

    SpeechSplit successfully converts intended speech aspects without altering the others, whereas AUTOVC only converts timbre.

  • Takeaways & Limitations

    Different information bottlenecks can force separate channels to carry information unavailable elsewhere, providing a generic approach to disentanglement.

  • Takeaways & Limitations

    AUTOVC’s bottleneck removes timbre but leaves content, rhythm, and pitch lumped together, motivating SpeechSplit’s additional channels.

Abstract

from arXiv · show

Speech information can be roughly decomposed into four components: language content, timbre, pitch, and rhythm. Obtaining disentangled representations of these components is useful in many speech analysis and generation applications. Recently, state-of-the-art voice conversion systems have led to speech representations that can disentangle speaker-dependent and independent information. However, these systems can only disentangle timbre, while information about pitch, rhythm and content is still mixed together. Further disentangling the remaining speech components is an under-determined problem in the absence of explicit annotations for each component, which are difficult and expensive to obtain. In this paper, we propose SpeechSplit, which can blindly decompose speech into its four components by introducing three carefully designed information bottlenecks. SpeechSplit is among the first algorithms that can separately perform style transfer on timbre, pitch and rhythm without text labels. Our code is publicly available at https://github.com/auspicious3000/SpeechSplit.

1. Introduction

Speech can be decomposed into content, timbre, pitch, and rhythm, but existing systems mainly separate timbre. SpeechSplit addresses the remaining underdetermined decomposition without text labels using three information bottlenecks.

  • Motivation: Speech comprises language content, speaker timbre, pitch, and rhythm, which support analysis and generation applications.Content can be transcribed; timbre reflects speaker identity; pitch and rhythm contribute to prosody.
  • Motivation: Existing voice conversion systems disentangle timbre but leave content, pitch, and rhythm mixed together.Their converted speech therefore changes mainly in timbre while pitch contour and rhythm remain largely unchanged.
  • Research gap: Obtaining annotations for content, rhythm, and pitch is difficult or expensive, making unsupervised decomposition necessary.Rhythm requires transcriptions, while pitch contours themselves remain entangled with rhythm.
  • Approach: SpeechSplit blindly decomposes speech into content, timbre, pitch, and rhythm with an encoder-decoder containing three distinct information bottlenecks.The bottlenecks constrain representation dimensions and randomly resample along time, encouraging different channels to carry different components.
  • Contribution: SpeechSplit transfers different speech aspects to different styles without relying on text transcriptions.Its channels are designed so that one passes content, one rhythm, and one pitch information, while timbre is separately handled in generation.

2. Related Work

Prior speech decomposition work includes source-filter signal-processing methods, timbre-focused voice conversion, and prosody-oriented TTS systems. These approaches provide relevant components but do not establish the paper’s full unsupervised decomposition setting.

  • The Source-Filter Model: Source-filter approaches decompose speech into pitch-bearing source and content-bearing filter components using signal-processing or probabilistic methods.Examples include linear predictive coding, cepstral analysis, temporally stable power spectral analysis, and probabilistic approaches.
  • The Source-Filter Model: Source-filter methods do not consider speech prosody aspects.This leaves rhythm and pitch-related prosodic structure insufficiently addressed in that line of work.
  • Voice Conversion: Voice conversion research has used VAEs and GANs to disentangle speaker timbre from speech.VAE-VC directly applies a variational autoencoder whose encoder produces a speaker-independent content representation.
  • Prosody Disentanglement: Prosody-disentanglement TTS systems extract or condition on latent and explicit prosody representations to generate expressive speech.The cited systems target prosody separation through prosody encoders, labels, or hierarchical encoder-decoder structures.
  • Background example: Figure 1 compares spectrograms and pitch contours for two utterances of the same sentence, highlighting formants and pitch tones.The figure links spectral formant structure with timbre-related variation and pitch contours with pitch-related variation.

3. Background: Information in Speech

Speech information includes content, rhythm, pitch, and timbre, each reflected in distinct but partly entangled acoustic patterns. The background explains how timing, pitch contours, and formants represent these components.

  • Overview: Figure 1 uses spectrograms and pitch contours from two utterances of “Please call Stella” to illustrate speech components.Formants are highlighted in the spectrogram, while pitch tones are highlighted in the pitch contour.
  • Rhythm: Rhythm is represented by the lengths of spectrogram segments corresponding to words or syllables.Longer segments indicate slower speaking, whereas shorter segments indicate faster speaking.
  • Pitch: Pitch represents intonation through pitch targets such as rises, falls, high tones, and low tones.Pitch targets are reflected in pitch contours, which also contain rhythm and speaker-identity information.
  • Timbre: Timbre is reflected by formant frequencies, the resonant frequency components of the vocal tract.Higher or lower formant frequency ranges correspond to brighter or darker voice qualities in the illustrated examples.
  • Content: Content is represented by phone-specific formant patterns that can remain similar across different timbre ranges.The highlighted “ea” phone has a shared formant pattern even when its formant frequencies differ between speakers.

4. SPEECHSPLIT

SPEECHSPLIT extends information-bottleneck autoencoding with three heterogeneous encoder channels to decompose speech into content, rhythm, and pitch while using speaker identity for timbre. Its bottlenecks and input-specific transformations are designed so each channel preserves information unavailable elsewhere, yielding disentangled codes under stated assumptions.

  • Framework: SPEECHSPLIT models speech as content, rhythm, pitch, and timbre, representing the first three with Zc, Zr, and Zf while speaker identity is separately supplied to the decoder.The generative formulation assumes S = gs(C, R, F, V) and U = gu(V), with content including residual information outside rhythm, pitch, and timbre.
  • Relation to AUTOVC: The framework generalizes AUTOVC’s timbre disentanglement: AUTOVC removes timbre but leaves content, rhythm, and pitch lumped together, whereas SPEECHSPLIT targets all remaining components.SPEECHSPLIT uses three encoder channels to further decompose the information that AUTOVC’s single bottleneck cannot separate.
  • Framework: Three heterogeneous bottlenecks use speech inputs for content and rhythm, normalized pitch contours for pitch, and random resampling to remove rhythm information from selected channels.The content and pitch encoders apply random resampling, while all encoders also have physical bottlenecks at their outputs.
  • Disentanglement rationale: The rhythm encoder preserves complete rhythm because it alone accesses all rhythm information, whereas the other encoders receive randomly resampled rhythm.Its bottleneck therefore prioritizes retaining rhythm and discarding information that the other channels can supply.
  • Disentanglement rationale: After rhythm is isolated, the content encoder must preserve content because the pitch encoder cannot access it; the pitch encoder then preserves pitch.The argument assumes the other components can be supplied through separate channels and the relevant bottlenecks are binding.
  • Formal result: With appropriately set bottlenecks and sufficient representation power, a global optimum of the reconstruction objective produces the disentangled representation.The formal claim is contingent on relatively stringent assumptions, including properties of random resampling and the normalized pitch contour.

5. Experiments

Experiments on VCTK evaluate SpeechSplit through single- and multi-aspect conversions, subjective judgments, pitch metrics, visualization, and bottleneck tests. The results support separate conversion of rhythm, pitch, and timbre, while wider bottlenecks impair disentanglement.

  • Experimental setup: SpeechSplit was trained on 20 VCTK speakers and compared with AUTOVC, using parallel speech pairs only for testing.Training used about 15 minutes per speaker, ADAM, batch size 16, and 800k steps.
  • Conversion visualization: Single-aspect visualizations show rhythm-only conversion matching target timing, pitch-only conversion matching target pitch, and timbre-only conversion matching target formants.The other aspects remain aligned with the source in each corresponding single-aspect conversion.
  • Subjective evaluation: Subjective conversion rates are high for the corresponding converted aspect and low for the others, whereas AUTOVC primarily converts timbre.This pattern holds across single- and double-aspect conversions evaluated against source and target references.
  • Subjective quality: MOS decreases as more aspects are converted, and pitch conversion receives higher MOS than timbre or rhythm conversion.The authors characterize timbre and rhythm conversion as more challenging tasks.
  • Objective evaluation: SPEECHSPLIT achieves a GPE of 1.04%, a VDE of 8.14%, and an FFE of 8.86% for pitch-only conversion.The authors describe these results as showing effectiveness in pitch disentanglement, while cautioning that datasets differ from the reference results.
  • Rhythm conversion: Rhythm conversion stretches short utterances or squeezes long utterances into the target syllable slots, suggesting a decoder mechanism that fills rhythm-defined blanks with content and pitch.The observations also suggest stable anchoring of content and pitch to the appropriate slots.
  • Bottleneck analysis: When rhythm or content bottlenecks are too wide, subjective conversion patterns degrade, with rhythm leakage especially causing rhythm-only conversion to alter all aspects.Removing rhythm produces a blank output, while removing content produces slots without informative spectral shape.

6. Conclusion

The conclusion presents SpeechSplit’s multiple information bottlenecks as enabling strong disentanglement and motivating a broader design principle for disentangled representation learning. It also identifies bottleneck refinement and low-resource speech processing as future directions.

  • Conclusion: SpeechSplit demonstrates strong disentanglement through multiple carefully designed information bottlenecks.The conclusion frames the result as the central demonstrated capability of the model.
  • Takeaways: Physical hidden-representation dimensions can effectively limit information flow.This is presented as the first takeaway from the experiments.
  • Takeaways: When an information bottleneck is binding, an autoencoder passes information that other channels cannot provide.The conclusion states this as a general observation supported by the model’s behavior.
  • Takeaways: Multiple channels with different bottlenecks can complete disentanglement even when one component, such as random resampling, is only partially disentangled.The authors propose this as a generic approach to disentanglement.
  • Future directions: Future work will refine SpeechSplit’s bottleneck design using more information-theory-guided approaches and explore low-resource speech processing applications.These are the specific future directions stated in the conclusion.

A. Proof to Theorem 1

The proof establishes that achieving zero reconstruction loss forces the rhythm, content, and pitch codes to satisfy their target information constraints. It combines contradiction arguments showing that violating any constraint prevents reconstruction of the corresponding speech aspect.

  • Lemma 1.1 states that, under Theorem 1’s assumptions, the global minimum of equation (4) is 0.
  • If the rhythm code fails equation (2), the reconstructed speech cannot recover rhythm and therefore cannot reconstruct the full speech, contradicting zero optimal loss.
  • If the content code fails equation (2) after the rhythm constraint holds, the reconstruction cannot recover content and therefore cannot reconstruct the full speech.
  • If the pitch code fails equation (2) after the rhythm and content constraints hold, the reconstruction cannot recover pitch and therefore cannot reconstruct the full speech.
  • Combining Lemmas 1.3, 1.4, and 1.5 implies Theorem 1, while a decoder construction achieves 0 reconstruction loss in equation (4).

B. Additional Implementation Details

The implementation uses 80-dimensional melspectrograms and converts speaker-normalized pitch contours into 256-bin one-hot representations.

  • Input and output spectrograms are 80-dimensional melspectrograms computed with 64 ms frames and 16 ms hops.
  • Pitch contours are extracted per speaker, normalized by their mean and four times their standard deviation, and roughly constrained to 0-1.
  • The normalized pitch range is quantized into 256 bins and represented as one-hot vectors.

B.2. Information Bottleneck Implementations

SpeechSplit restricts information flow through random resampling and physical-dimension constraints, while alignment procedures support rhythm, pitch, content, and timbre disentanglement. The section also provides bottleneck-tuning diagnostics and evaluation-sample selection procedures.

  • Information Bottleneck Implementations: SpeechSplit restricts information flow using random resampling and constraints on code dimensions, including frequency- and time-domain downsampling.
  • Information Bottleneck Implementations: Randomly sized 19–32-frame segments are linearly resampled with factors from 0.5 to 1.5, using shared factors for the content and pitch encoders.
  • Information Bottleneck Implementations: During pitch conversion, a rhythm-only conversion restores temporal alignment after the pitch contour is replaced by another utterance.
  • Information Bottleneck Implementations: The pitch-contour rhythm-only variant retains rhythm and pitch encoders while removing the content encoder.
  • Information Bottleneck Implementations: Bottleneck dimensions are tuned by zeroing each code and checking whether the resulting spectrogram matches the expected diagnostic pattern.A zero rhythm code should produce an almost blank output, while a zero content code should produce uninformative spectral slots.
  • Information Bottleneck Implementations: Pitch and rhythm evaluations use 20 speaker-diverse parallel pairs selected for perceptible pitch or large duration differences, respectively.

C. Additional Experiment Results

Additional experiments test whether random resampling removes only part of rhythm information. Reconstructions with different resampling factors remain similar and recover the ground truth reasonably well.

  • Additional Experiment Results: An autoencoder experiment tests whether random resampling removes only part of rhythm information rather than eliminating it completely.
  • Additional Experiment Results: Two reconstructions with different randomly drawn resampling factors are very alike despite using the same ground-truth utterance.
  • Additional Experiment Results: Both reconstructions recover the ground-truth speech decently, with only minor blurring.

C.2. Do Rhythm Labels Exist?

The study tests whether short-time energy and unvoiced-voiced labels can replace SpeechSplit’s rhythm code. Although both candidates align timing, they fail to preserve syllable content because they lack anchor information, whereas SpeechSplit’s rhythm code supports successful conversion.

  • Candidate rhythm labels: Short-time energy and UV labels are tested as candidate rhythm labels alongside SpeechSplit’s rhythm encoder.Short-time energy uses a moving average of squared waveform values, while UV labels indicate whether each pitch-contour frame is voiced.
  • Candidate rhythm labels: Both candidate labels align the converted speech temporally with the target, suggesting that syllable timing is transferred.The comparison uses rhythm-only conversion on two utterances with target rhythm information substituted into each system.
  • Candidate rhythm labels: The candidate conversions place incorrect content within syllables despite their temporal alignment.Inspection of the formant structure shows that the content within each syllable is completely incorrect.
  • Why candidates fail: The candidates locate syllable boundaries but lack anchor information specifying what content belongs in each temporal blank.This explains why the conversion algorithms put the wrong content into correctly located syllable intervals.
  • Why candidates fail: SpeechSplit’s triple information bottleneck produces a rhythm code that supplies anchor information while excluding excessive content, enabling successful rhythm conversion.The passage characterizes obtaining such a rhythm label as nontrivial.

C.3. Additional Conversion Spectrograms

Additional spectrograms evaluate SpeechSplit across single- and multi-aspect conversions. The results show intended aspects matching the target while remaining aspects retain source characteristics, and converting all three aspects yields speech very similar to the target.

  • Additional spectrograms: The additional spectrograms cover two utterances and display rhythm+pitch, rhythm+timbre, and pitch+timbre conversions.The utterances are “One showing mainly red and yellow” and “Six spoons of fresh snow peas.”
  • Conversion results: SpeechSplit converts the intended aspects to match the target while keeping the remaining aspects matched to the source.This pattern is reported consistently with the earlier spectrogram results.
  • Conversion results: When all three aspects are converted, the resulting speech becomes very similar to the target speech.
Loading 2004.11284v6…