PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Contrastive Representation Learning Based Voice Conversion for Electrolaryngeal Speech Enhancement

Ding Ma, Jinyi Mi, Jiajun He, Kazuhiro Kobayashi, Naoki Nishio, Tomoki Toda

arXiv:2610.17716v1cs.SD

TL;DR

EL speech differs substantially from normal speech, and existing multimodal approaches may rely on extra modalities at runtime. MCRL-VC fuses speech and text representations, then transfers them to a speech-only EL2SP model. The authors report that it outperforms speech-only and multimodal-based VC baselines.

  • Problem

    EL–normal speech mismatches hinder robust intermediate representations for EL2SP, while multimodal approaches may add complexity, data requirements, or runtime modality needs.

    Source: §I · p. 1

  • Method

    MCRL-VC applies cross-modal contrastive learning to fuse speech and text representations, then cross-stage contrastive learning to transfer them to speech-only EL2SP.

    Source: §I · p. 2, A. Framework Overview · p. 2

  • Results

    The proposed method outperformed both speech-only and multimodal-based VC baselines, and stage-wise variant analysis confirmed each stage's contribution.

    Source: §I · p. 2, §V · p. 6, B. Stage-wise Validation of… · p. 6

  • Takeaways & Limitations

    MCRL-VC is designed to improve EL2SP while enabling inference without text input.

    Source: §V · p. 6

Abstract

from arXiv · show

Electrolaryngeal (EL) speech suffers from severe acoustic mismatches with normal speech, resulting in degraded naturalness and intelligibility. These mismatches also make sequence-to-sequence voice conversion struggle to learn robust intermediate representations, hindering EL-speech-to-normal-speech conversion (EL2SP) performance. To address this, we propose a novel multimodal contrastive representation learning framework to 1) achieve richer multimodal-representation learning, and 2) facilitate the transferability of multimodal representations to speech-only EL2SP. First, EL speech and text representations are fused to learn multimodal representations for target speech generation, during which cross-modal contrastive learning is employed to encourage a semantically consistent speech--text representation space. Next, the fused representations are treated as a fixed target to guide speech-only representation learning through cross-stage contrastive learning, enabling multimodal knowledge to be transferred to the EL2SP task while preserving architectural and inference simplicity. Experimental results demonstrate that the proposed method consistently outperforms both the speech-only and multimodal-based baselines.

I. INTRODUCTION

EL speech and normal speech differ substantially, while existing multimodal approaches may require added modalities at inference. MCRL-VC uses contrastive learning to fuse speech and text representations and transfer multimodal knowledge to EL2SP.

  • EL speech differs from normal speech in acoustic cues and temporal structure, contributing to reduced naturalness, clarity, and communication burdens.Artificial excitation lacks dynamic vocal-fold characteristics, and compensatory speaking styles can affect duration and temporal structure.Source: §I · p. 1
  • Multimodal fusion can integrate text with speech representations, but often increases model complexity and data needs or relies on extra modalities at runtime.Source: §I · p. 1
  • MCRL-VC uses cross-modal contrastive learning for speech–text representation fusion, then cross-stage contrastive learning to transfer fused representations to EL2SP.The framework is intended to address multimodal fusion and knowledge transfer while preserving a simple architecture and inference pipeline.Source: §I · pp. 1–2
  • The experimental results show that the proposed method outperforms both a speech-only method and a multimodal method with the same modalities.The authors demonstrate the effectiveness of each stage in the proposed method.Source: §I · p. 2

II. PROPOSED METHOD

MCRL-VC transfers multimodal knowledge into a speech-only EL2SP model through two stages: speech–text fusion followed by speech-encoder guidance.

A. Framework Overview

  • Stage 1 aligns EL speech and text representations in a shared semantic space and fuses them for mapping toward the speech decoder.Source: A. Framework Overview · p. 2
  • In Stage 2, the Stage 1 integrated representations guide a speech encoder to enable speech-only reconstruction.Source: A. Framework Overview · p. 2

B. Stage 1: Cross-modal Text-guided Representation Learning

Stage 1 combines EL speech and text representations for target speech generation, using cross-modal contrastive learning to align paired modalities in a shared semantic space.

B. Stage 1: Cross-modal Text-guided Representation Learning

C. Stage 2: Cross-stage Multimodal Representation Transfer

Stage 2 removes the text encoder and trains a speech-only EL2SP model, using the frozen text-enhanced representation from Stage 1 as a contrastive teacher.

C. Stage 2: Cross-stage Multimodal Representation Transfer

D. Training Loss

Training combines the standard reconstruction, stop-token, and guided-attention losses with contrastive objectives assigned to the two stages.

D. Training Loss

  • Stage 1 adds the cross-modal contrastive loss, and Stage 2 adds the cross-stage contrastive loss to the model’s standard training objectives.The standard objectives are L1 target-speech reconstruction, weighted binary cross-entropy for stop-token prediction, and guided attention for diagonal sequence alignment.Source: D. Training Loss · p. 3
  • The weights α and β control the contributions of the two contrastive objectives.Source: D. Training Loss · p. 3

A. System Building

The evaluation compares MCRL-VC with speech-only and multimodal baselines, including a reconstruction-trained variant that omits cross-stage contrastive learning.

A. System Building

  • MCRL-VC is compared with S-VC and MR-VC, plus MCRL-VC-Rec, which retains first-stage contrastive learning but uses MR-VC’s reconstruction training in Stage 2.Source: A. System Building · pp. 3–4
  • S-VC uses VC pretraining and two-stage fine-tuning, with synthetic parallel data in its first fine-tuning stage and original speech data in the second.Source: A. System Building · pp. 3–4
  • MR-VC trains a speech-text joint network with cross-attention for normal-speech generation, then removes the text encoder for speech-only reconstruction.Source: A. System Building · p. 4
  • MCRL-VC uses the same training-data settings and pretrained VC and normal-TTS modules as MR-VC.Source: A. System Building · p. 4

B. Datasets

The study uses JSUT for VC pretraining and evaluates EL2SP on two small-scale patient datasets: one parallel and one semi-parallel. The datasets also use external parallel speech data during specified training stages.

1) TTS Database:

  • JSUT contributes 7,296 training utterances (about 10 hours), with its transcriptions and generated speech data used in first-stage training.The corpus was used for VC pretraining.Source: 1) TTS Database · p. 4

2) EL2SP Datasets:

  • Patient 1 is a roughly 15-minute parallel corpus of 300 EL and normal utterances recorded from the same patient before and after laryngectomy.Source: 2) EL2SP Datasets · p. 4
  • Patient 2 is semi-parallel: the patient's 200 EL utterances are paired with 413 normal utterances from a healthy speaker because the patient's preoperative speech was unavailable.This dataset simulates the practical case in which a patient's own normal speech cannot be obtained.Source: 2) EL2SP Datasets · p. 4
  • Each dataset reserves 20 parallel utterances for development and 40 for testing; remaining utterances are used for training.Source: 2) EL2SP Datasets · p. 4
  • External parallel speech data (7,296 utterances) supplements original parallel pairs during first-stage fine-tuning or joint-network training; final-stage training reuses only original parallel data.For Patient 2, generated EL speech counterparts allow all 353 normal utterances to be used in those earlier stages.Source: 2) EL2SP Datasets · p. 4

C. Implementations

The systems use Transformer-based speech and text processing and are evaluated with objective and subjective measures. MCRL-VC improves over MR-VC on both patient datasets, with stage-wise contrastive learning associated with stronger representation alignment and overall conversion performance.

C. Implementations

  • The implementation uses six-layer Transformer blocks with four attention heads, 24 kHz audio, and 80-dimensional mel-filterbank features.The cross-attention block also uses four heads; output waveforms are reconstructed with speech-dependent Parallel WaveGAN vocoders.Source: C. Implementations · p. 4

D. Evaluation Metrics

  • Evaluation includes MCD, CER, SECS, and predicted MOS; subjective NMOS is assessed by 26 native Japanese listeners on randomly selected utterances.The table captions report objective and subjective results for each patient, with 95% confidence intervals for NMOS.Source: D. Evaluation Metrics · p. 4, Table I · p. 5, Table II · p. 5

A. Comparison of MCRL-VC and Baselines

B. Stage-wise Validation of Contrastive Learning

Stage-wise comparisons associate cross-modal contrastive learning with stronger multimodal fusion and cross-stage contrastive learning with improved transfer to speech-only conversion. The conclusion reports that MCRL-VC outperforms speech-only and multimodal baselines without requiring text at inference.

B. Stage-wise Validation of Contrastive Learning

  • MCRL-VC-Rec outperforms MR-VC, while MCRL-VC outperforms MCRL-VC-Rec on three of four metrics, with a slight SSQA trade-off.The comparisons support the contribution of first-stage cross-modal contrastive learning and second-stage cross-stage contrastive learning, respectively.Source: B. Stage-wise Validation of… · p. 5
  • MCRL-VC shows the strongest diagonal dominance and lowest off-diagonal similarities, indicating closer alignment between second-stage speech and first-stage fused representations.MCRL-VC-Rec also has a clearer diagonal pattern and weaker off-diagonal similarities than MR-VC.Source: B. Stage-wise Validation of… · pp. 5–6

V. CONCLUSION

  • The conclusion reports that MCRL-VC outperforms speech-only and multimodal VC baselines and performs EL2SP without text input.Future work targets additional modalities and low-latency EL2SP systems.Source: §V · p. 6
Loading 2610.17716v1…