Iterative Self-Learning for Expressive Text-to-Speech Synthesis

Nicholas Sanders; Gustav Eje Henter; Simon King; Korin Richmond

Source-linked AI summary

Iterative Self-Learning for Expressive Text-to-Speech Synthesis

Nicholas Sanders, Gustav Eje Henter, Simon King, Korin Richmond

arXiv:2608.15910v1Aug 16, 2026eess.AScs.CLcs.SD

TL;DR

Expressive TTS needs costly human-labeled emotion and prominence data, while prior semi-supervised methods do not address scarce discrete expressive labels. This paper introduces iterative self-learning with classifier-free label recovery, finding improved pseudo-label accuracy and generally better downstream controllability across both tasks in bounded low-resource settings.

Problem
Prior semi-supervised TTS methods do not address the scarcity of discrete symbolic expressive labels needed for explicit controllability.
Method
The framework repeatedly uses a single evolving TTS model to recover labels for unlabeled speech, then retrains on seed and pseudo-labeled data.
Results
Across prominence and emotion, iterative refinement improved pseudo-label accuracy beyond single-pass pseudo-labeling and generally improved adherence, synthesis metrics, and listener preferences.
Takeaways & Limitations
Pseudo-label accuracy was a useful indicator of downstream expressive controllability, but effective self-learning depended on task and training configuration.
Takeaways & Limitations
ISL was evaluated only with a Matcha-TTS backbone, so whether its self-learning dynamics persist at larger model scales remains open.

Abstract

from arXiv · show

Expressive text-to-speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.

I. INTRODUCTION · II. RELATED WORK · A. Iterative Self-Learning and Pseudo-Labeling

The paper addresses expressive-label scarcity in controllable TTS by applying iterative self-learning with Invert-Classify to unlabeled speech. It evaluates whether repeated pseudo-labeling and retraining improve expressive label quality and synthesis for word-level prominence and utterance-level emotion.

I. INTRODUCTION: Explicit expressive representations offer interpretable control over attributes such as prominence and emotion, but their use is constrained by scarce large-scale labeled speech corpora.Expressive information is often entangled with linguistic, paralinguistic, and speaker-specific acoustic correlates.
I. INTRODUCTION: Human annotation of expressive categories is expensive and difficult to perform consistently, motivating automated approaches that otherwise require separate classifiers or representation-adaptation stages.The passage identifies emotion and prominence as categories requiring human perceptual judgments.
I. INTRODUCTION: ISL leverages unlabeled data by repeatedly pseudo-labeling it with a model trained on limited labeled data, retraining on the combined corpus, and reusing the updated model.In speech processing, this procedure is commonly called Iterative Pseudo-Labeling (IPL).
I. INTRODUCTION: Unlike discriminative tasks, expressive TTS must jointly maintain expressive-label adherence and overall synthesis quality, because these objectives can decouple.The paper therefore motivates repeated pseudo-labeling and retraining as the model evolves.
I. INTRODUCTION: The proposed framework trains a TTS model on a small annotated seed, freezes it for Invert-Classify pseudo-labeling, then retrains on seed and pseudo-labeled data without external classifiers or pretrained models.The cycle is iterated to address expressive-label scarcity.
A. Iterative Self-Learning and Pseudo-Labeling: ASR IPL commonly uses cyclical pseudo-labeling and retraining to avoid premature convergence to initial-seed biases, but heterogeneous data and heuristic selection can cause degradation.Related implementations may reinitialize models or retain only high-confidence unlabeled examples.
A. Iterative Self-Learning and Pseudo-Labeling: Iterative self-learning must manage label-noise dynamics because deep networks learn simple patterns early and memorize noise later, while expressive-label noise can reflect task subjectivity.Inter-annotator agreement can vary substantially across expressive traits.

B. Semi-Supervised TTS

Prior semi-supervised TTS methods mainly use unlabeled speech to learn acoustic representations or address missing speech, transcripts, and paired data rather than refining explicit expressive labels. This work addresses that gap by embedding Invert-Classify in an iterative loop that progressively improves synthesis and label recovery.

Prior semi-supervised TTS: Semi-supervised TTS has generally leveraged unpaired speech for internal representation learning instead of iterative label refinement.This contrasts with semi-supervised ASR, where iterative pseudo-label refinement and cycling are common.
Prior semi-supervised TTS: VQ-based methods extract discrete acoustic units from untranscribed speech to enable TTS training without aligned text–speech supervision.Tu et al. use a VQ-VAE trained for speech reconstruction, while QS-TTS advances the same general strategy through distillation.
Prior semi-supervised TTS: Other methods address scarcity from the opposite direction by generating synthetic speech targets for text corpora with a teacher model.StrawNet assumes speech, rather than transcripts, is scarce for a given text corpus.
Controllability gap: These approaches do not solve the scarcity of discrete symbolic expressive labels needed for explicit controllability.A partially supervised VAE encourages latent variables to correlate with affect and speaking rate, but retains implicit rather than symbolic control.
Proposed approach: The proposed framework embeds Invert-Classify into an Iterative Self-Learning loop, using an evolving generative TTS teacher to refine synthesis and labeling quality.Invert-Classify previously recovered explicit expressive labels only through single-pass pseudo-labeling; ISL extends it with iterative refinement.

III. METHODS … C. Modeling Expressive Representations

The methods introduce an iterative self-learning framework built around a single evolving Matcha-TTS model and gradient-based pseudo-labeling, with expressive conditioning implemented at word and utterance levels. The pipeline repeatedly self-labels unlabeled speech and retrains on the resulting data while using efficient flow matching and standard phoneme processing.

A. Iterative Self-Learning (ISL) Framework: ISL uses a single evolving model that repeatedly self-labels and retrains on its own predictions.The framework progressively refines both the generative model and training labels through cyclical interaction.
A. Iterative Self-Learning (ISL) Framework: Initialization trains seed model M0 for 100 epochs exclusively on the small ground-truth labeled dataset before unlabeled-data exposure.The current model is then frozen for Invert-Classify pseudo-labeling of unlabeled speech.
B. Matcha-TTS Backbone: The acoustic backbone is Matcha-TTS, selected to support spectrogram generative modeling with iterative self-learning and gradient-based pseudo-labeling.The choice avoids additional pretrained dependencies and training complexity associated with larger pretrained components and speech/language-model backbones.
B. Matcha-TTS Backbone: Matcha-TTS improves inference efficiency over earlier stochastic diffusion models by using Optimal Transport Conditional Flow Matching to learn an ODE transport trajectory.The trajectory maps noise to data and is close to a straight line, reducing the need for many inference steps.
B. Matcha-TTS Backbone: Text processing follows the original released setup, using phonemizer with an espeak-ng frontend for grapheme-to-phoneme conversion.
C. Modeling Expressive Representations: Word-level prominence is represented by binary <PROM> and <NOTPROM> tags interleaved with phoneme tokens at word boundaries.The prominence tags are learned as trainable embeddings in a separate embedding table, decoupling phonetic and prominence representations.
C. Modeling Expressive Representations: Utterance-level emotion is represented by concatenating a learned target-emotion embedding to the text encoder outputs.This follows a standard conditioning approach in emotional TTS.

D. Generative Modeling via OT-CFM · E. Gradient-Based Pseudo-Labeling

The framework uses OT-CFM to model mel-spectrogram generation while stabilizing duration and encoder predictions with auxiliary losses. For pseudo-labeling, Invert-Classify freezes the seed model and recovers missing label representations through gradient-based inversion under deterministic flow-matching conditions.

D. Generative Modeling via OT-CFM: OT-CFM learns a vector field transporting Gaussian noise to the mel-spectrogram data distribution along nearly straight trajectories.The model is trained with Optimal Transport Conditional Flow Matching.
D. Generative Modeling via OT-CFM: Auxiliary losses Ldur and Lenc stabilize duration predictions and encoder representations, with λdur and λenc weighting the combined fusion loss.The combined objective is used for both model training and inversion.
D. Generative Modeling via OT-CFM: The combined objective serves both as the training objective and as the inversion loss.This links generative-model training directly to pseudo-label recovery.
E. Gradient-Based Pseudo-Labeling: Invert-Classify freezes a model trained on seed labeled data and pseudo-labels unlabeled samples through a two-step process.It is a classifier-free pseudo-labeling method for generative models.
E. Gradient-Based Pseudo-Labeling: The missing label representation is initialized as the mean of all possible label embeddings and updated by gradient descent using LTotal.The update constitutes inversion with respect to the same loss used to train the seed model.
E. Gradient-Based Pseudo-Labeling: Inversion fixes t to a constant value and reuses one noise tensor x0 for each input across all inversion steps.This deterministic environment prevents stochastic sampling from continually shifting the objective and blocking convergence.
E. Gradient-Based Pseudo-Labeling: All experiments use t = 0.9, which produced greater conditioning-label gradient magnitudes for the evaluated tasks and datasets.Fixing t makes ∇LTotal primarily reflect conditioning labels rather than stochastic sampling variance.

F. Dataset Selection and Retraining Strategy … B. Low-Resource Data Partitioning

The study combines Select and Retrain to prevent iterative pseudo-label errors from accumulating with carefully structured datasets and low-resource partitions. ESD uses stratified speaker–emotion sampling, while Naver-Prosody uses lexical-group sampling to preserve contrastive prominence structure.

F. Dataset Selection and Retraining Strategy: Iterative pseudo-labeling requires balancing label accuracy against the number of training epochs per iteration, because insufficient training can harm synthesis and indefinite looping can reinforce errors.Pseudo-label quality affects realization of target expressive traits, while epoch choices affect convergence and generalization.
F. Dataset Selection and Retraining Strategy: Select and Retrain identifies iteration i∗ with the highest held-out validation pseudo-label accuracy, then retrains a new model from scratch on the fixed dataset D(i∗)comb.This decouples improved labels from the model state accumulated during iterative training.
A. Datasets: ESD contains approximately 14.5 hours from 10 native English speakers across Neutral, Happy, Angry, Sad, and Surprise.The English subset uses parallel lexical content across emotion categories to minimize lexical bias, with official train-test splits and 16 kHz-to-22.05 kHz resampling.
A. Datasets: Naver-Prosody contains 26.7 hours and approximately 36,600 utterances from one female American English speaker, with binary prominence targets derived from contrastive focus groups.Interrogatives are excluded, leaving groups of four parallel utterances per text.
B. Low-Resource Data Partitioning: ESD uses stratified random sampling by Speaker ID and Emotion Class, applying the target percentage separately within each stratum.This guarantees full coverage of speakers and emotion categories in extreme low-resource settings.
B. Low-Resource Data Partitioning: Naver-Prosody samples lexical groups rather than individual utterances, so a “1% split” includes 1% of available quadruplets.Every selected text retains all four prominence-placement variations, preserving the dataset’s discriminative structure at low data volumes.

C. Model Configuration and Hyperparameters … A. Generalizability and Iterative Dynamics

The paper specifies reproducible TTS and inversion configurations, evaluates pseudo-labeling, synthesis, and expressive-label adherence, and tests ISL across expressive tasks, data-scarcity levels, and training durations. Generalizability experiments use matched partitions and iteration-wise validation to examine label-noise propagation and convergence–stability trade-offs.

C. Model Configuration and Hyperparameters: TTS models use Adam at 4 × 10−4 without weight decay, gradient clipping at 5.0, batch size 32, and 100 seed-training epochs.Final Select-and-Retrain models run for 1000 epochs; synthesis uses Euler ODE sampling with temperature 1.0 and 40 steps.
C. Model Configuration and Hyperparameters: Inversion updates emotion embeddings for 200 steps at 0.01 and prominence embeddings for 100 steps at 10−4, both with batch size 32.A fixed HiFi-GAN V1 vocoder is trained for 500,000 steps using 80-bin mel-spectrograms at 22.05 kHz.
D. Evaluation Metrics: Evaluation covers pseudo-label accuracy, objective synthesis quality, and objective and subjective expressive-label adherence.These axes jointly assess label recovery, generated speech quality, and conditioning fidelity.
D. Evaluation Metrics: Emotion pseudo-labeling is evaluated with Macro F1, averaging per-class F1 equally across five emotion classes.Prominence uses Binary F1 for the <PROM> class, emphasizing precision and recall in prosodic-stress detection.
D. Evaluation Metrics: Synthesis quality uses TTSDS2, a normalized 2-Wasserstein distance between synthesized and reference speech across multiple feature spaces.Reported factors include General Quality and Prosody.
D. Evaluation Metrics: Emotion adherence is measured with emotion2vec_plus_large Macro F1 between intended and predicted synthesized-audio emotion labels.Subjective expressivity uses A/B preference tests with 30 Prolific listeners and 30 questions per task.
V. EXPERIMENTS: The experiments study Generalizability and Iterative Dynamics, then Performance Limit & Initialization, using identical splits across both datasets to simulate data scarcity.ISL is applied to utterance-level emotion and word-level prominence labels.
A. Generalizability and Iterative Dynamics: Both tasks use 0.5%, 1%, 5%, and 20% partitions, while training-duration conditions test noise propagation through Conservative (3 Epochs), Standard (10 Epochs), and Saturation (50 Epochs) schedules.The 0.5% ESD condition contains exactly 50 samples, and each iteration is evaluated on held-out validation data using Macro F1 for ESD and Binary F1 for Naver.

B. Performance Limit and Initialization · VI. RESULTS · A. Impact of Training Duration

The study separates pseudo-label quality from training duration by comparing iterative self-learning conditions with single-pass and fully supervised references. Across prominence and emotion, 10 epochs per iteration provides the most consistent stability, whereas 50 epochs causes model collapse.

B. Performance Limit and Initialization: The experiment isolates pseudo-labeled-data quality from training-iteration quantity by retraining fresh models from random initialization for 1000 epochs.The models use datasets derived from Experiment 1 under a Select-and-Retrain methodology.
B. Performance Limit and Initialization: The optimal ISL condition uses pseudo-labels from iteration i∗, where pseudo-label accuracy is highest after self-refinement.This condition represents the high-quality, self-refined data setting.
B. Performance Limit and Initialization: The single-pass control uses base-model outputs from iteration 0, while the 100% GT reference is a fully supervised performance ceiling.These resulting models support objective synthesis evaluation and subjective listening tests.
VI. RESULTS: The self-learning loop’s stability is evaluated through pseudo-label accuracy on held-out validation sets across all data partitions.Figures 2 and 3 visualize the dynamics under varying training intensities.
A. Impact of Training Duration: ≈0.45 to ≈0.24: F1-score collapses in the 1% Prominence split under Saturation (50 Epochs).Longer training between pseudo-labeling phases degrades pseudo-label accuracy across both tasks.
A. Impact of Training Duration: All Emotion splits show model collapse under Saturation (50 Epochs), with pseudo-labeling accuracy falling significantly below the seed model’s initial accuracy.This is a consistent failure mode across the emotion task.
A. Impact of Training Duration: The Standard (10 Epochs) and Conservative (3 Epochs) settings improve stability by limiting training per iteration.The 10-epoch condition is consistently most robust, while 3 epochs occasionally achieves higher late-stage performance.
A. Impact of Training Duration: 10 epochs per iteration is identified as the most consistent optimal trade-off between convergence speed, noise resistance, and pseudo-label accuracy.The 3-epoch setting can outperform it at late stages in some prominence figures, including Figs. 3c and 3b.

B. Generalizability and Task-Dependent Learning Dynamics · C. Impact of Data Scarcity

ISL learns expressivity at task-dependent rates: emotion benefits rapidly from self-training, whereas prominence improves more gradually and may degrade after a local peak. Its advantages are strongest in severely data-scarce settings, require sufficient supervised signal, and diminish when supervision is already adequate.

B. Generalizability and Task-Dependent Learning Dynamics: Emotion control gains most of its Macro F1-Score improvement in the first self-training pass, rising from ≈0.35 to over ≈0.45 on the 5% split.Subsequent iterations provide only marginal stability gains and saturate quickly.
B. Generalizability and Task-Dependent Learning Dynamics: Word-level prominence improves more gradually, requiring additional iterations before pseudo-labeling performance stabilizes.In both tasks, performance can plateau or degrade after reaching a local pseudo-labeling accuracy peak.
B. Generalizability and Task-Dependent Learning Dynamics: Task-specific pseudo-labeling trajectories determine how many ISL iterations are effective and which checkpoints are selected for final synthesis experiments.The differing trajectories motivate different checkpoint choices for the synthesis analyses.
C. Impact of Data Scarcity: ISL is most effective when supervised data are scarce, with its efficacy increasing as supervised data availability decreases.The results establish an inverse relationship between supervised data availability and self-learning-loop efficacy.
C. Impact of Data Scarcity: In the 0.5% Emotion split, pseudo-label accuracy remains flat, indicating that bootstrapping requires a minimum threshold of supervised signal.This marks a lower bound on ISL efficacy under extreme data scarcity.
C. Impact of Data Scarcity: At 20% supervised data, ISL benefits disappear, yielding diminishing returns once the supervised signal supports robust generalization.The subsequent synthesis analysis therefore focuses on Emotion at 1% and 5%, and Prominence at 0.5% and 1%.

D. Objective Synthesis Quality and Expressive Label Adherence · 1) Expressive Label Adherence:

ISL-trained models generally improve expressive label adherence over single-pass controls, with especially clear gains for emotion and one exception in the lowest-resource prominence condition. For prominence, listening tests provide the primary adherence measure because no reliable pretrained classifier was available.

1) Expressive Label Adherence:: ISL models generally outperform Single-Pass Control models in expressive label adherence, except under the lowest-resource prominence condition.
1) Expressive Label Adherence:: 29.87 emo2vec F1 is achieved by 1% ISL versus 26.58 for the Control on emotion label adherence.
1) Expressive Label Adherence:: 49.58 emo2vec F1 is achieved by 5% ISL versus 39.38 for the Control, widening the emotion adherence gap.
1) Expressive Label Adherence:: Improved pseudo-label accuracy translates into better expressive label adherence in final synthesis.
1) Expressive Label Adherence:: 49.58 emo2vec F1 from 5% ISL exceeds 45.95 from 100% Ground Truth, though domain effects and synthetic-speech uncertainty motivate relying on listening tests and other objective results.
1) Expressive Label Adherence:: Prominence adherence is evaluated perceptually in the listening study because no reliable pretrained classifier was available.Results are reported in Section VI-E.

2) Prosody, Quality, and Speaker Similarity: · E. Subjective Evaluation

ISL generally maintains or improves objective synthesis quality while improving expressive control across low-resource prominence and emotion tasks. Subjectively, ISL is preferred over single-pass controls at more resourced splits, but gaps remain against fully supervised references in some conditions.

2) Prosody, Quality, and Speaker Similarity:: ISL generally maintains or improves objective synthesis quality beyond expressive label adherence.This applies to objective synthesis metrics across the evaluated expressive tasks.
2) Prosody, Quality, and Speaker Similarity:: For prominence, ISL improves overall TTSDS scores relative to single-pass controls at 0.5% and 1% data splits (91.37 vs. 93.15; 91.57 vs. 92.66).Improvements are particularly evident in intelligibility-related metrics, while prosody remains broadly comparable.
2) Prosody, Quality, and Speaker Similarity:: The prominence TTSDS Pitch metric is anomalous because DIO voicing detection fails on low-F0 frames in natural speech.The analysis found that 97.7% of natural GT utterances contain voiced regions below DIO’s 72 Hz pitch floor, motivating pYIN-based F0 RMSE reporting.
2) Prosody, Quality, and Speaker Similarity:: For emotion, ISL improves overall TTSDS scores relative to single-pass controls at 1% and 5% data splits (88.70 vs. 89.90; 89.18 vs. 89.85).Gains are most pronounced in prosody-related metrics and accompany improved Emo2vec classification performance.
E. Subjective Evaluation: ISL is preferred over single-pass control for expressivity at more resourced splits in prominence and emotion (NP: 0.678 ± 0.078; ESD 5%: 0.633 ± 0.037).These preferences indicate that improved pseudo-labeling translates to perceptually stronger expressive synthesis.
E. Subjective Evaluation: At the most data-scarce conditions, ISL-trained models are preferred for prominence and emotion, but a gap from fully supervised training remains for emotion.The supplied passage truncates the prominence comparison before reporting its complete result.
E. Subjective Evaluation: The 100% GT reference is preferred over 5% ISL for emotion (0.633 ± 0.042) but tied with 1% ISL for prominence (0.629±0.132).Anchor comparisons show Vocoded GT preferred over the fully supervised reference for prominence (0.617 ± 0.110) and tied for emotion (0.533 ± 0.062).

VII. DISCUSSION · VIII. LIMITATIONS · IX. CONCLUSION

The discussion and conclusion identify ISL as effective for expressive-label scarcity within a bounded low-resource regime, while highlighting limitations in supervision, representation, backbone coverage, and comparison scope. Future work should extend expressive representations and architectures and improve robustness against noisy pseudo-labels.

VII. DISCUSSION: VII. DISCUSSION: Pseudo-label accuracy on held-out validation data generally tracked expressive label adherence, objective synthesis metrics, and subjective preferences across emotion and prominence.The magnitude of these gains depended on the expressive task and the amount of ground-truth seed data.
VII. DISCUSSION: VII. DISCUSSION: ISL was most effective in a bounded low-resource regime, with diminishing returns or degradation from larger seed sets and insufficient bootstrap supervision from extremely small ones.Degradation with larger seed sets was consistent with overfitting to self-generated labels.
VII. DISCUSSION: VII. DISCUSSION: Binary prominence labels do not capture the variability of their acoustic realization across pitch, duration, intensity, voice quality, context, and transcriber judgments.The passage motivates alternative approaches that represent prominence more flexibly.
VIII. LIMITATIONS: VIII. LIMITATIONS: ISL was evaluated only with a Matcha-TTS backbone, although its gradient-based pseudo-labeling mechanism has also been demonstrated with regression-based and flow-matching generative models.The study therefore does not directly establish performance across larger diffusion-based or LLM-based TTS systems.
VIII. LIMITATIONS: VIII. LIMITATIONS: The study does not directly compare self-contained gradient-based pseudo-labeling with external classifier-based pipelines because their architectures, supervision requirements, and classifier choices vary.Fair comparison requires matched supervision budgets across expressive tasks.
IX. CONCLUSION: IX. CONCLUSION: The paper extends inversion-based pseudo-labeling into iterative self-learning and evaluates it for word-level prominence and utterance-level emotion.The framework targets expressive label scarcity in TTS using gradient-based pseudo-labeling.
IX. CONCLUSION: IX. CONCLUSION: Iterative refinement improved expressive pseudo-label accuracy beyond single-pass pseudo-labeling, but excessive retraining degraded accuracy and too little seed data prevented reliable supervision.Effectiveness depended on the expressive task and ISL training configuration, and was greatest within a bounded low-resource regime.
IX. CONCLUSION: IX. CONCLUSION: Future work should examine continuous prominence, affective control spaces, larger generative architectures, confidence-based selection, adaptive stopping, and selective re-initialization.These techniques may limit noisy pseudo-label propagation and improve robustness.