PaperlayerView exact revision on arXiv

Source-linked AI summary

ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers

Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa-Johnson, Shiyu Chang

arXiv:2204.09224v2cs.SDcs.AIeess.AS

TL;DR

Content-focused speech tasks need speaker-disentangled representations, but removing speaker information can also damage content. CONTENTVEC adapts HuBERT with mechanisms that regularize teacher labels and learned representations, and evaluations report consistent advantages across content-related speech applications.

  • Problem

    Content-focused speech tasks need speaker variation separated from content, but speaker removal can cause content loss whose cost outweighs disentanglement benefits.

  • Method

    CONTENTVEC adapts HuBERT by combining teacher disentanglement, student disentanglement, and speaker conditioning.

  • Results

    CONTENTVEC achieves a consistent performance advantage over baseline speech representations across content-related speech processing applications.

  • Takeaways & Limitations

    Speaker disentanglement can improve content-related speech processing while retaining content information, although the paper reports slight content loss.

  • Takeaways & Limitations

    CONTENTVEC still has slight content loss and lacks a method for selecting hyperparameters.

Abstract

from arXiv · show

Self-supervised learning in speech involves training a speech representation network on a large-scale unannotated speech corpus, and then applying the learned representations to downstream tasks. Since the majority of the downstream tasks of SSL learning in speech largely focus on the content information in speech, the most desirable speech representations should be able to disentangle unwanted variations, such as speaker variations, from the content. However, disentangling speakers is very challenging, because removing the speaker information could easily result in a loss of content as well, and the damage of the latter usually far outweighs the benefit of the former. In this paper, we propose a new SSL method that can achieve speaker disentanglement without severe loss of content. Our approach is adapted from the HuBERT framework, and incorporates disentangling mechanisms to regularize both the teacher labels and the learned representations. We evaluate the benefit of speaker disentanglement on a set of content-related downstream tasks, and observe a consistent and notable performance advantage of our speaker-disentangled representations.

1. Introduction

Speech SSL learns representations from unannotated data for downstream tasks, but content-focused applications need speaker variation disentangled without sacrificing content. CONTENTVEC adapts HuBERT with speaker-disentanglement mechanisms and shows consistent advantages on content-related applications.

  • Motivation: Speech SSL trains representations on large unannotated corpora, reducing downstream reliance on large labeled datasets.The learned representations are transferred to downstream tasks with limited annotated data.
  • Motivation: Content-focused speech tasks need representations that separate speech content from interfering speaker variations.Existing widely used representations often retain substantial speaker information; HuBERT reaches up to 81.4% speaker-identification accuracy on SUPERB.
  • Problem: Removing speaker variation without text annotations can also remove content, whose loss often outweighs the benefit of speaker disentanglement.This trade-off makes speaker disentanglement particularly challenging for content-related downstream tasks.
  • Approach: CONTENTVEC adapts HuBERT’s teacher–student masked-prediction framework to combine speaker disentanglement with content preservation.The approach is motivated by HuBERT students that can improve on imperfect teacher representations, including in content preservation.
  • Approach: CONTENTVEC incorporates teacher disentanglement, student disentanglement, and speaker conditioning to regulate speaker-information flow.Teacher disentanglement removes speaker information from labels, student disentanglement enforces speaker invariance, and conditioning supplies speaker information to prediction.
  • Findings: Extensive evaluation finds a consistent performance advantage over baseline speech representations on content-related applications.The findings support speaker-disentangled representations for more targeted downstream information.

2. Related Work

Prior work studies speaker–content disentanglement extensively in voice conversion and develops multiple self-supervised speech representation-learning paradigms. These approaches include analysis–synthesis models, generative and masked-prediction methods, multi-task learning, and semi-supervised learning.

  • Voice Conversion: Voice conversion commonly follows an analysis–synthesis framework that learns speaker-independent content representations before synthesizing speech with speaker variation.Research targets improved linguistic representations, better speaker-variation injection, or both.
  • Voice Conversion: Voice-conversion methods pursue disentanglement through variational autoencoders, auxiliary speaker classifiers, output classification, adversarial architectures, and bottleneck tuning.Examples include VAE-VC, speaker-classifier approaches, ACVAE-VC, StarGAN variants, CycleGAN variants, and AutoVC.
  • Self-Supervised Learning in Speech: Self-supervised speech representation learning generally encodes speech into contextual representations and applies pretext tasks to extract content information.The related work distinguishes generative learning from masked-prediction approaches.
  • Self-Supervised Learning in Speech: Generative SSL methods include autoregressive predictive coding and Mockingjay, while HuBERT predicts masked frames using k-means-quantized targets.ILS-SSL extends HuBERT with masked-prediction loss on intermediate layers.
  • Self-Supervised Learning in Speech: Other speech SSL approaches combine multiple tasks or use both labeled and unlabeled data.These methods broaden the training objectives or supervision sources beyond a single self-supervised task.

3. Approach

CONTENTVEC adapts HuBERT’s masked-prediction framework to preserve content while reducing speaker information. It combines teacher disentanglement, student speaker-invariance regularization, and speaker conditioning in the predictor.

  • Problem formulation: CONTENTVEC learns representations that should preserve speech content while remaining invariant across speaker variations.The approach treats content preservation and speaker invariance as competing goals.
  • General framework: HuBERT’s framework masks the input, predicts teacher labels from masked representations, and trains the representation network and predictor jointly.The teacher generates labels from unmasked speech, while the student processes the masked utterance.
  • Disentanglement in teachers: Teacher disentanglement converts utterances to one speaker, extracts representations with pretrained HuBERT, and quantizes them into discrete labels.This removes much speaker information from teacher labels before masked prediction.
  • Disentanglement in teachers: Because voice conversion can lose content, CONTENTVEC uses the converted representations to train better students rather than applying them directly downstream.The paper identifies non-negligible content loss for some speakers as a limitation of the teacher representation.
  • Disentanglement in students: Student disentanglement applies speaker-altering transformations and a symmetric contrastive loss to penalize differences between the resulting representations.The transformations are applied before masking, and the contrastive loss can be imposed at an intermediate or final layer.
  • Speaker conditioning: Speaker conditioning feeds speaker embeddings to the predictor, while the final objective combines masked prediction and contrastive losses.This prevents residual speaker information in teacher labels from forcing the student representations to encode the same information.

4. Experiments

The experiments evaluate CONTENTVEC against speech-representation baselines across zero-shot content probes, language modeling, supervised content tasks, speaker and accent classification, voice conversion, and ablations. Across these evaluations, CONTENTVEC improves content-related performance while reducing speaker information, and its three disentangling mechanisms contribute to the observed behavior.

  • Zero-shot Content Probe: CONTENTVEC achieves a consistent advantage across all four zero-shot probing metrics, with the largest gains on phonetic-level ABX(w) and ABX(a) tasks.Gains are smaller for lexical and syntactic tasks, where language-model quality also affects performance.
  • Language Modeling: CONTENTVEC achieves significantly lower PPX and slightly lower VERT than all baselines, indicating improved language-model correctness and speech-generation quality.
  • SUPERB Experiments: CONTENTVEC generally outperforms HUBERT-ITER and HUBERT on SUPERB content and semantic tasks under matched teacher-cluster and training-iteration settings.The comparison uses continuous representations, with the representation networks frozen during downstream-task training.
  • Speaker & Accent Classification: CONTENTVEC sharply reduces SID and accent-classification accuracy, including a SID reduction of up to 36% relative to HUBERT-ITER.The results indicate that speaker disentanglement also reduces accent information to some degree.
  • Voice Conversion: CONTENTVEC further improves average d-vector cosine similarity between converted speech and target speakers, supporting stronger speaker disentanglement in voice conversion.
  • Ablation Studies: Removing any of the three modules makes performance significantly worse, while alternative contrastive-loss positions and weights remain competitive.The evaluated modules are teacher disentanglement, student disentanglement, and speaker conditioning; the contrastive-loss sweep uses weights from 1e-6 to 5e-5.

5. Conclusions

CONTENTVEC removes speaker information while preserving content by building on HuBERT with three disentanglement components. Experiments support the importance of all three modules and show benefits across content-related speech tasks, while slight content loss and absent hyperparameter selection remain limitations.

  • CONTENTVEC targets speaker-information removal while preventing content-information loss.
  • The method adds disentanglement in teachers, disentanglement in students, and teacher conditioning of the predictor to HuBERT.
  • Ablation results confirm that all three modules are essential to CONTENTVEC's success.
  • Successful speaker disentanglement improves performance across a wide range of content-related speech-processing tasks.
  • The remaining limitations are slight content loss and the lack of a hyperparameter-selection method.

A.1. Visualizing Speech Generation Cluster Sequences

The appendix compares speaker-related cluster sequences for matched male and female speech, using ranked label occurrence ratios and dynamic time warping. CONTENTVEC produces lower average framewise label distance, indicating that gender information is hardly preserved in its features.

  • The comparison ranks k-means labels by their female-to-male occurrence ratio before analyzing matched male and female speech.
  • Dynamic time warping aligns converted and original utterances because the converted audio has a slight speech-rate mismatch.
  • CONTENTVEC achieves lower average L0 distance between ranked label sequences, indicating that gender information is hardly preserved in its features.

A.2. ASR Transcriptions of Example Generated Speech

The appendix examines transcripts from speech resynthesized with prompts using HUBERT-ITER and CONTENTVEC. CONTENTVEC appears better at reconstructing prompts, but neither representation generates meaningful long sentences.

  • The comparison uses the first three seconds of test-clean Librispeech utterances to generate example transcripts from prompts.
  • CONTENTVEC appears better at reconstructing the prompts than HUBERT-ITER.
  • Both models generate meaningful speech within a few words after the prompt.
  • Neither model generates meaningful long sentences.

A.3. Voice Converter Quality

The voice-converter ablation tests whether converter quality affects CONTENTVEC by replacing the standard converter with one trained for fewer steps. The compromised converter degrades the best-layer ABX(w), ABX(a), and PNMI results.

  • The ablation replaces a 40,000-step voice converter with a compromised version trained for 10,000 steps.
  • With the compromised converter, best-layer ABX(w), ABX(a), and PNMI degrade to 6.05, 7.78, and 0.5616, respectively.

A.4. Contrastive Loss on Multiple Layers

The ablation tests whether spreading contrastive loss across multiple representation layers improves CONTENTVEC. It does not: the multi-layer configuration is slightly worse than the original model.

  • A.4. Contrastive Loss on Multiple Layers: The ablation imposes contrastive loss on the last, last-but-third, and last-but-seventh layers, reducing each loss weight to 1/3.This keeps the overall loss scale consistent while testing multi-layer contrastive supervision.
  • A.4. Contrastive Loss on Multiple Layers: 5.19, 6.53, and 0.5684 are the best-layer ABX(w), ABX(a), and PNMI results for the multi-layer model.The best layer for all three metrics is layer 12.
  • A.4. Contrastive Loss on Multiple Layers: The multi-layer model is slightly worse than the original CONTENTVEC model.The reported results provide no evidence that imposing contrastive loss on multiple layers improves CONTENTVEC.

B.1. CONTENTVEC Implementation and Training

CONTENTVEC uses the HuBERT speech-representation architecture and output-layer features, with masking restricted to masked prediction. Its predictor injects speaker information through conditional normalization, and training uses voice-conversion-derived teacher labels.

  • Architecture: CONTENTVEC follows HuBERT with seven temporal-convolution blocks and 12 transformer layers of model dimension 768.Each layer is independently dropped with probability 0.05.
  • Feature extraction: The output-layer representation is selected as the CONTENTVEC feature because speaker disentanglement is optimal there.The masking probability is 0.08, matching the stated Wav2Vec 2.0 masking strategy.
  • Training objectives: Masking is applied only to the masked prediction loss, L_pred, and not to the contrastive loss, L_cont.The masks of the two transformation paths are the same.
  • Predictor: The three-layer predictor injects speaker information through conditional layer normalization whose scale and bias depend on speaker embeddings.The frame-masking scheme and prediction-logit generation match HuBERT.
  • Training: Training runs for 100k steps on 36 GPUs, with the best model selected by validation masked-prediction loss.Each GPU processes at most 76 seconds of audio per batch, and training takes about 19 hours.
  • Teacher-label generation: Teacher labels are generated with a voice converter retrained on a 200-speaker LibriSpeech subset.The implementation and hyperparameters otherwise follow the publicly available system unless explicitly changed.

B.2. Baselines and Dataset

The evaluation compares CONTENTVEC with Wav2Vec 2.0, HuBERT, and an iterative HuBERT baseline on zero-shot content-probing and generation-related tasks. Models use LibriSpeech data, standardized representation quantization, and shared language-model and synthesis procedures where applicable.

  • Baselines: Wav2Vec 2.0 uses its 14th-layer representation, while HuBERT uses its sixth-layer representation.These are the selected baseline layers for evaluation.
  • Baselines: HUBERT-ITER is trained with the same pretrained HuBERT teachers as CONTENTVEC but without voice conversion, controlling for iterative training.Layer eight is selected after teacher-quality evaluation.
  • Dataset: CONTENTVEC and all baselines are trained on the full 960-hour LibriSpeech dataset.Teacher-quality evaluation uses the dev-clean and dev-other partitions.
  • Representation processing: Discrete representations are quantized to 100 k-means clusters, except for SUPERB, which uses 500 clusters.Continuous representations are used when the evaluation task does not require discreteness.
  • Content probing: Zero-shot probing tasks are selected from the Zero-Resource Speech Challenges because they require close alignment between discrete representations and phonetic content.The same transformer language-model architecture and hyperparameter setting are used where language modeling is required.
  • Language modeling: Language-model generation is evaluated across temperatures using perplexity, VERT, and AUC for the quality-variety trade-off.VERT is the geometric mean of self-BLEU and auto-BLEU, while lower AUC indicates closer proximity to the ground-truth anchor point.
  • Voice conversion: Voice-conversion comparisons use CONTENTVEC, HuBERT, and HUBERT-ITER continuous units with the same voice-conversion model and resynthesis procedure.The 200-speaker corpus is split into approximately 70% training, 10% development, and 20% test utterances per speaker.
Loading 2204.09224v2…