Source-linked AI summary
MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation
Szu-Chi Chen, I-Ning Tsai, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee
TL;DR
S2ST systems often preserve semantic content but lose expressive cues such as laughter and crying, despite their pragmatic importance. MoVE combines scalable expressive-data synthesis with AudioLLM adaptation and dynamically blended vocalization experts, achieving state-of-the-art expressive fidelity with 30 minutes of curated data reaching 95% of full-data emotional fidelity.
Problem
Recent S2ST systems struggle to preserve expressive acoustic cues and non-verbal vocalizations such as laughter and crying, which convey pragmatic intent.
Method
MoVE combines a scalable expressive-data synthesis pipeline with pretrained AudioLLM adaptation and a soft-weighting router that blends specialized vocalization adapters.
Results
30 minutes of curated data achieves 95% of full-data emotional fidelity, while MoVE establishes state-of-the-art expressive S2ST performance.
Takeaways & Limitations
MoVE demonstrates that pretrained AudioLLMs can transfer emotions and non-verbal vocalizations with high fidelity and strong data efficiency in S2ST.
Abstract
from arXiv · showhide
Recent Speech-to-Speech Translation (S2ST) systems achieve strong semantic accuracy yet consistently strip away non-verbal vocalizations (NVs), such as laughter and crying that convey pragmatic intent, which severely limits real-world utility. We address this via three contributions. First, we propose a synthesis pipeline for building scalable expressive datasets to overcome the data scarcity limitation. Second, we propose MoVE, a Mixture-of-LoRA-Experts architecture with expressive-specialized adapters and a soft-weighting router that blends experts for capturing hybrid expressive states. Third, we show pretrained AudioLLMs enable striking data efficiency: 30 minutes of curated data is enough for strong performance. On English-Chinese S2ST, while comparing with strong baselines, MoVE reproduces target NVs in 76% of cases and achieves the highest human-rated naturalness and emotional fidelity among all compared systems, where existing S2ST systems preserve at most 14% of NVs.
1 National Taiwan University, Taipei, Taiwan 2 NVIDIA, Taiwan · 1. Introduction
The introduction identifies lost prosody, emotion, and non-verbal vocalizations as major pragmatic weaknesses in otherwise semantically accurate S2ST. It presents scalable expressive-data synthesis and pretrained AudioLLM adaptation, including architecture changes for multi-emotion modeling, as the proposed remedies.
- 1. Introduction: S2ST combines ASR, MT, and TTS to enable direct cross-lingual vocal interaction, but losing prosody and emotional nuance can create serious pragmatic biases.
- 1. Introduction: Recent S2ST systems achieve high semantic accuracy yet struggle with acoustic expressiveness and preserving non-verbal vocalizations such as laughter and crying.
- 1. Introduction: High-quality speech corpora containing authentic non-verbal vocalizations are scarce, contaminated, and rarely suitable for expressive S2ST training in structure, quality, and quantity.
- 1. Introduction: Training expressive S2ST is difficult because the model must jointly handle ASR, MT, TTS, cross-linguistic semantic alignment, and diverse expressive acoustic preservation.
- 1. Introduction: The paper proposes a scalable expressive dataset synthesis pipeline and pretrained AudioLLM fine-tuning, while studying data scale and multi-emotion architectural modifications.
- 1. Introduction: The work introduces automated generation-selection for expressive S2ST data spanning affective states and specific non-verbal vocalizations, including laughter.
- 1. Introduction: The paper reports that curated data achieves 95% of full-data emotional fidelity without compromising semantic translation accuracy.
2. Methodology
MoVE training is grounded in a scalable pipeline for synthesizing expressive English–Chinese speech-translation pairs from parallel text. Its first stage curates diverse acoustic prompts to avoid narrow emotional stereotypes and preserve a broad affective manifold.
- Expressive data synthesis: The synthesis pipeline uses parallel English–Chinese text from GigaSpeech and GigaST to generate expressive speech-translation pairs through an emotion-adaptive process.The process is designed as a scalable foundation for MoVE training.
- Expressive prompt curation: Expressive prompt curation aggregates Happy, Sad, and Angry samples from CREMA-D, MSP-IMPROV, and IEMOCAP to maintain broad affective coverage.The curation step is intended to prevent narrow emotional stereotypes and preserve a broad, continuous affective manifold.
2. Emotion-Adaptive Synthesis via Attribute Decou-
The expressive data pipeline uses emotion-adaptive synthesis and quality filtering, while MoVE preserves pretrained AudioLLM capabilities through specialized LoRA experts, soft token-level routing, and expressive detokenization. Training independently specializes five experts before optimizing their unified router on the full dataset.
- Emotion-Adaptive Synthesis via Attribute Decoupling: Identity and expression are disentangled by conditioning IndexTTS2 on curated NV prompts, improving target paralinguistic induction without sacrificing speaker fidelity.A single acoustic reference supplies speaker identity and emotional prosody for standard affective states, while curated NV prompts address limited extreme-NV prompt diversity.
- Emotion-Adaptive Synthesis via Attribute Decoupling: The synthesis dataset focuses on Happy, Sad, and Angry as primary affective extrema, excluding acoustically ambiguous states and treating neutral variation through arousal-based sampling.States in the high-valence/low-arousal quadrant are excluded because their acoustic proximity to neutral speech creates classification ambiguity.
- Automated Quality Assurance and S2ST Pairing: Quality assurance sequentially trims silence, discards outputs shorter than 0.5 seconds, and verifies normalized transcripts with Whisper-small using WER ≤0.5.The lenient WER threshold accommodates NV vocalizations, which lack consistent lexical transcriptions and naturally elevate WER.
- MoVE Architecture: MoVE freezes the pretrained AudioLLM and injects 5 parallel LoRA adapters specializing in Happy, Sad, Angry, Laughing, and Crying across transformer layers.Independent low-rank subspaces target attention and feed-forward gating projections to avoid feature interference while preserving ASR, TTS, and cross-lingual semantics.
- MoVE Architecture: A dynamic Softmax router assigns continuous token-level weights to the frozen base and five experts, enabling fine-grained blending of hybrid expressive states.For each speech token, the router computes expert contributions using lightweight linear functions; N = 5 denotes the distinct acoustic manifolds.
- Training: MoVE fine-tunes the expressive detokenizer and trains experts in two stages: independent specialization followed by end-to-end router optimization on the full dataset.The router learns blending from the final language-modeling loss without explicit labels, while the detokenizer is trained on IndexTTS2-synthesized NV speech.
3. Experiments and Analysis
The experiments evaluate MoVE across semantic preservation, emotional fidelity, and human perception, while isolating dataset quality, architectural design, data efficiency, and routing behavior. MoVE consistently outperforms relevant baselines, and its router learns affective structure without explicit emotion labels.
- Evaluation Setup: The evaluation separates semantic translation, objective emotional fidelity, and subjective human perception, including ASR-BLEU and exact NV Match Accuracy.Tests use 1000 English-Chinese pairs from CVSS-T and assess whether perceived NVs exactly align with the source.
- Main Results: MoVE establishes a new state-of-the-art result against training-free baselines, alternative training datasets, its single-LoRA baseline, and a cascaded reference system.The comparison uses similarly sized training subsets of 50–100 hours where applicable.
- Training Dataset Comparison: Even the 50h subset of MoVE’s expressive corpus overwhelmingly surpasses the 100h SynStard subset and 67h SeamlessAlignExpressive on objective metrics.This comparison uses single-LoRA models to isolate expressive-corpus quality.
- MoVE Architectural Comparison: MoVE surpasses the identical-data single-LoRA baseline across all evaluation metrics, while achieving the highest Emotion SMOS, Naturalness MOS, A/B preference, and NV Match Accuracy.Both models are trained on the same 100h dataset, showing that expressive specialization does not compromise semantic accuracy.
- Router Analysis: 63.68% router alignment accuracy emerges without discrete emotion labels, while overlaps such as Sad–Cry and Happy–Laugh reflect hybrid expressiveness enabled by soft weighting.The router is trained only through final language-modeling loss and learns from latent linguistic and acoustic cues.
4. Conclusions
The paper addresses the expressive gap in S2ST with a scalable expressive-data curation pipeline and MoVE, which achieves state-of-the-art emotion and NV transfer fidelity using as little as 30 minutes of curated data. The method does not yet cover all emotions and NV categories.
- The paper proposes a scalable, expressive data curation pipeline for training and demonstrates its superiority over other datasets.
- Leveraging pretrained AudioLLM priors, MoVE achieves state-of-the-art fidelity in transferring emotions and NVs.
- 30 minutes of curated data can unlock deep expressive capabilities with MoVE.
- The method does not cover all emotions and NV categories.
A. Subjective Human Evaluation Protocol
The subjective evaluation used five bilingual evaluators and randomized presentation across a 30-utterance English–Chinese test set. It combined MOS ratings, pairwise architectural preferences, and source-anchored NV matching for laughing and crying while speaking.
- Evaluation Participants and Randomization: Five proficient English–Chinese bilingual evaluators rated a 30-utterance test set spanning six categories, with set and model orders independently randomized per subject.The test set contained six categories × five utterances.
- Instructions and Calibration: Before each phase, bilingual instructions defined 1–5 rating scales, Laughing and Crying while speaking categories, and the Speech Overlap principle.Speech Overlap required NVs to co-occur with linguistic content.
- Phase 1: MOS Multi-Stimulus Test: Phase 1 presented six anonymized model outputs for independent 1–5 Emotion Similarity and Naturalness ratings, plus perceived-NV selections.The outputs were MoVE, cascaded, SeamlessExpressive, Kimi-Audio, SeamlessM4T, and gpt-4o-audio.
- Phase 2: A/B Pairwise Preference Test: Phase 2 compared MoVE with the single-LoRA baseline in randomized A/B pairs, allowing Model A, Model B, or Tie / Similar judgments and NV cross-validation.Tie / Similar was restricted to perceptually indistinguishable cases.
- NV Match Accuracy: NV Match Accuracy counted a hit only when a model’s perceived NV label exactly matched the source-side annotation, averaging accuracies across laughing and crying subsets.Standard errors over the five evaluators were reported in Table 1.