PaperlayerView exact revision on arXiv

Source-linked AI summary

Articulatory Source-Filter TTS: Physically Grounded Control through Vocal Tract Kinematics

Jesuraj Bandekar, Shinji Watanabe, Prasanta Kumar Ghosh

arXiv:2610.00735v1eess.AS

TL;DR

Neural TTS systems can deliver high acoustic quality while offering little interpretable control over the vocal-tract filter. This paper grounds a controllable source–filter TTS model in AAI-generated articulatory trajectories and reports quality competitive with similarly sized baselines, with a modest spectral-fidelity cost.

  • Problem

    Neural TTS offers little interpretable control over the vocal-tract filter, while existing source–filter approaches lack a physically grounded articulatory parameterisation.

    Source: §I · p. 1

  • Method

    The model uses AAI-supervised articulatory trajectories to condition the filter and an OT-CFM module to refine the source conditioned on predicted source, filter, and speaker identity.

    Source: §I · p. 1, 3) Source Refinement via OT-CFM · p. 5

  • Results

    The model achieves intelligibility and naturalness competitive with similar-capacity baselines at a modest spectral-fidelity cost, with WER 5.53 versus XTTS’s 5.52.

    Source: B. TTS Quality and Model Capabilities · p. 10, §VII · p. 12

  • Takeaways & Limitations

    Source–filter recombination keeps F0 tied to the source speaker while spectral characteristics follow the filter speaker, and trajectory edits support accent modification.

    Source: §I · p. 2

    The reported synthesis quality involves a modest spectral-fidelity cost relative to baselines.

    Source: §VII · p. 12

Abstract

from arXiv · show

Modern neural text-to-speech (TTS) systems achieve remarkable acoustic fidelity but act as black boxes, offering little interpretable control over the vocal tract filter. We propose a controllable source-filter TTS architecture grounded in articulatory kinematics. An Acoustic-to-Articulatory Inversion (AAI) model, enhanced by large-scale pretrained representations, generates kinematic pseudo-trajectories for a large TTS corpus. These trajectories condition the filter response, while predicted pitch and energy contours parameterise the glottal source. Source and filter are predicted independently, and the source is refined by an Optimal Transport Conditional Flow Matching (OT-CFM) module before recombination into the final spectrogram. Our model achieves intelligibility and naturalness competitive with similarly sized baselines, with only a modest spectral fidelity cost in exchange for explicit control. Evaluations reveal clear source-filter disentanglement, enabling stable prosodic scaling and cross-speaker source/filter recombination, where F0 remains tied to the source speaker while vocal-tract characteristics follow the filter speaker. Finally, direct manipulation of articulatory trajectories enables fine-grained control, such as accent modification, offering a new direction for interpretable speech synthesis. Audio samples: https://coding-phoenix-12.github.io/ArticulatorySFTTS/

I. INTRODUCTION

Neural TTS offers strong acoustic quality but little interpretable control of the vocal tract, and existing source–filter models do not ground the filter in physical articulatory parameters. This work proposes text-driven synthesis with an articulatory-kinematics handle on the filter, alongside a separately controlled glottal source.

I. INTRODUCTION

  • Black-box TTS and existing source–filter systems lack a physically grounded vocal-tract control, limiting interpretable edits for applications such as dialect and accent modification.The proposed approach uses articulatory trajectories to ground the filter response in vocal-tract kinematics.Source: §I · p. 1
  • The source–filter decomposition separates pitch and energy in the glottal source from the vocal-tract filter, but neural TTS often replaces physical grounding with opaque acoustic representations.Physical synthesisers can be interpretable but may depend on hand-designed gestures or recorded audio rather than text.Source: §I · p. 1
  • The proposed text-driven TTS conditions its filter on AAI-supervised articulatory trajectories, providing kinematic-level control over vocal-tract response.The pseudo-labelling pipeline scales articulatory supervision from a small EMA corpus to hundreds of hours of unpaired TTS speech.Source: §I · pp. 1–2

A. Flow Matching based TTS

  • Flow matching transports prior noise toward speech features over time; conditional flow matching conditions that vector field on individual target samples, and TTS systems apply it to acoustic generation.The cited examples condition generation on linguistic encodings, durations, and speaker embeddings.Source: A. Flow Matching based TTS · p. 2

B. Controllable and Interpretable Source-Filter TTS

  • Existing controllable source–filter systems expose source attributes, but treat the filter as an abstract spectral object without physical correspondence to articulators.The paper contrasts these approaches with its articulatory control of the vocal-tract filter.Source: §I · p. 1, B. Controllable and Interpretable… · p. 2

C. Acoustic-to-Articulatory Inversion and Articulatory Speech Synthesis

Acoustic-to-articulatory inversion estimates vocal-tract movements from speech, while articulatory synthesis maps trajectories toward speech generation. The paper uses AAI to create pseudo-trajectories for training a text-driven source–filter TTS model.

C. Acoustic-to-Articulatory Inversion and Articulatory Speech Synthesis

  • AAI estimates physical articulator movements from recorded speech, whereas articulatory speech synthesis generates speech from articulatory trajectories.Together, these tasks model the relation between vocal-tract biomechanics and acoustic output.Source: C. Acoustic-to-Articulatory Inversion… · p. 2

III. PROPOSED METHODOLOGY

  • The proposed setup trains an AAI model on parallel EMA-speech data, then generates pseudo-trajectories across a large TTS corpus as targets for source–filter TTS training.The estimated kinematics cover the lips, jaw, and tongue; the paper later specifies 12-channel midsagittal trajectories.Source: §III · p. 3

A. Articulatory Pseudo-Target Generation via AAI

  • The AAI model uses a Transformer architecture with acoustic features from multilingual MMS-1B pretrained representations.The model is pretrained on over a thousand languages, providing phonetic variability relevant to inversion.Source: A. Articulatory Pseudo-Target… · p. 3
  • The AAI input concatenates MMS-1B features from layers 6, 24, and 36; at 16 kHz, the model produces feature frames at 50 Hz.Source: A. Articulatory Pseudo-Target… · p. 3
  • AAI training combines positional MSE with a Pearson-correlation loss to capture trajectory shape and movement over time.The final objective sums the two penalty terms.Source: A. Articulatory Pseudo-Target… · pp. 3–4

B. Controllable Source-Filter TTS Architecture

The architecture separates linguistic, source, and articulatory representations, then aligns and expands them to acoustic-frame length for independent source and filter prediction.

B. Controllable Source-Filter TTS Architecture

  • The TTS architecture builds on FastPitchFormant and, with the residual source target defined as audio minus the predicted filter response.The residual is used to train OT-CFM.Source: B. Controllable Source-Filter TTS… · p. 4

1) Linguistic Encoding and Length Regulation:

  • The Phoneme Encoder maps phonemes and a speaker embedding to linguistic hidden representations, which feed separate articulatory, source, and average-spectrogram streams.Separate streams support independent source and filter prediction.Source: 1) Linguistic Encoding and Length… · p. 4
  • MAS aligns the average-spectrogram stream to acoustic frames, producing phoneme durations used to train the Duration Predictor.The coarse average-spectrogram stream provides a decomposition-agnostic alignment prior.Source: 1) Linguistic Encoding and Length… · p. 4
  • Ground-truth durations expand the articulatory, source, and average-spectrogram streams by frame replication to match the target sequence length.Source: 1) Linguistic Encoding and Length… · p. 4

AEP, H∗

The predictors assign prosodic contours to the source and articulatory trajectories to the filter, which are estimated independently and combined in log-Mel space.

2) Source and Filter Prediction:

  • The PEP estimates fundamental-frequency and energy contours, supervised against ground-truth contours with combined MSE and Pearson-correlation objectives.These predicted prosodic contours expose the glottal source to direct control.Source: 2) Source and Filter Prediction · p. 4
  • During training, the SLP uses ground-truth pitch and energy so its source spectrogram is tied to the controllable prosodic contours.Source: 2) Source and Filter Prediction · p. 5
  • The FLP predicts the filter spectrogram from AAI-generated articulatory trajectories without pitch or energy, structurally encouraging source-filter separation.The source and filter predictors estimate isolated components from their respective physical inputs.Source: 2) Source and Filter Prediction · p. 5
  • The source and filter log-Mel spectrograms are added to form the audio prediction, which is optimized against the ground-truth audio spectrogram with MSE.Source: 2) Source and Filter Prediction · p. 5

3) Source Refinement via OT-CFM:

OT-CFM refines the source excitation to recover spectral and temporal detail, conditioning the refinement on predicted components and speaker identity.

3) Source Refinement via OT-CFM:

  • OT-CFM refines the MSE-trained source predictor’s over-smoothed spectra to restore fine spectral and temporal detail.The target source is obtained by subtracting the predicted filter response from the ground-truth audio spectrogram.Source: 3) Source Refinement via OT-CFM · p. 5
  • The refinement is conditioned on predicted source and filter components plus a speaker embedding, keeping it consistent with predicted content and speaker identity.Source: 3) Source Refinement via OT-CFM · p. 5
  • OT-CFM learns a vector field that maps prior noise to the target source over continuous time, with the interpolated state defined between them.The flow-matching loss regresses against the target vector field.Source: 3) Source Refinement via OT-CFM · p. 5

4) Training Objective and Inference:

The architecture is trained end-to-end with a summed multi-task objective and, at inference, predicts and combines the source and filter components before vocoding.

4) Training Objective and Inference:

  • The total training objective sums duration, prior, pitch-energy, articulatory-trajectory, Mel-spectrogram, and flow-matching losses.The prior is the MAS alignment prior loss.Source: 4) Training Objective and Inference · p. 5
  • At inference, predicted durations, articulatory kinematics, pitch, and energy replace ground-truth targets to generate the filter and source paths.OT-CFM samples prior noise and solves its learned ODE with an Euler solver to produce the refined source.Source: 4) Training Objective and Inference · p. 5
  • The predicted source and filter spectrograms are added, then passed to a pretrained vocoder for waveform reconstruction.Source: 4) Training Objective and Inference · p. 5
  • Source-side blocks receive prosodic attributes and filter-side blocks receive articulatory attributes, while speaker embedding and linguistic context are shared; this routing yields stable training and strong synthesis quality in practice.Source: 4) Training Objective and Inference · pp. 5–6

IV. EXPERIMENTAL SETUP

The experimental setup combines multiple EMA corpora for articulatory modeling with LibriTTS for synthesis, and evaluates both articulatory generalization and speech synthesis.

1) AAI:

  • Six EMA datasets provide synchronized audio and 12-channel trajectories for six articulators, with the corpus breakdown organized by language, speakers, and subset duration.The trajectories record 2D midsagittal coordinates.Source: 1) AAI · p. 6, Table II · p. 6
  • AAI generalization is tested on HPRC using seen speakers with unseen utterances and held-out speakers, isolating American English for the target TTS dialect.TTS evaluation uses LibriTTS test-clean and test-other; parallel VCTK utterances support cross-speaker control evaluation.Source: 1) AAI · p. 6

B. Model architecture and Training

  • The AAI backend uses a non-autoregressive encoder-decoder with four Feed-Forward Transformer blocks in both encoder and decoder, producing 12-channel EMA trajectories.The model has about 7.3M parameters.Source: 1) AAI · p. 6

1) AAI:

The setup aligns pretrained acoustic representations with articulatory targets, specifies the synthesis architecture and training, and compares it with AAI, TTS, controllable-synthesis, and ablation baselines.

1) AAI:

  • TERA, MMS-SSL, and ASR-finetuned MMS features are compared for AAI, with 50 Hz MMS features duplicated to align with 100 Hz kinematic trajectories.Kinematic channels are normalized independently; AAI training uses Adam at 1 × 10^-4 with validation-loss early stopping.Source: 1) AAI · p. 6

2) TTS:

  • The TTS system predicts pitch and energy alongside articulatory trajectories, then uses separate source and filter predictors and OT-CFM to refine the source before vocoding.At inference, the source and filter spectrograms are summed and synthesized by HiFi-GAN; the complete model has approximately 33M parameters.Source: 2) TTS · pp. 6–7

C. Baseline Models

  • Baselines include TERA AAI, YourTTS, XTTS, adapted Matcha-TTS, StyleTTS2, and HierSpeech++, alongside variants removing articulatory conditioning or OT-CFM.The no-OT-CFM variant uses the MSE-trained source predictor directly, and subjective MOS was not collected for it.Source: C. Baseline Models · p. 7

V. EVALUATION METRICS

Evaluation covers articulatory accuracy, speech quality, and explicit controllability, using objective metrics and subjective naturalness ratings.

A. Articulatory Trajectory Prediction Performance

  • Articulatory accuracy is measured by Pearson correlation on HPRC seen- and unseen-speaker data, for both acoustic-to-articulatory inversion and text-predicted trajectories.The AAI evaluation compares TERA, MMS-SSL, and MMS-FT representations; text-to-articulatory evaluation tests zero-shot physical accuracy against HPRC data.Source: A. Articulatory Trajectory Prediction… · pp. 7–8

B. Speech Synthesis and Model Evaluation

  • Speech synthesis is assessed using WER and CER, MCD, pitch and energy log-MSE and correlation, speaker-embedding cosine similarity, and naturalness measures.Naturalness uses MOS ratings from 20 listeners on 20 utterances per system and UTMOS over the full LibriTTS test sets.Source: B. Speech Synthesis and Model Evaluation · p. 8
  • Controllability and interpretability are additive scores based on explicitly manipulable attributes and exposed intermediate representations, respectively.The maximum scores are 3 for controllability and 5 for interpretability.Source: B. Speech Synthesis and Model Evaluation · p. 8

C. Explicit Control and Disentanglement Analysis

The reported text-to-articulatory result approaches a dedicated AAI baseline despite zero-shot cross-corpus evaluation. The section also evaluates prosodic and cross-speaker disentanglement and illustrates trajectory-based articulatory control.

C. Explicit Control and Disentanglement Analysis

  • Prosodic control is evaluated by scaling predicted pitch and energy from 0.7 to 1.3 and checking whether formants remain unperturbed.The comparison includes StyleTTS2 and HierSpeech++, and evaluates generated contours against equivalently scaled ground truth.Source: C. Explicit Control and… · p. 8
  • Cross-speaker disentanglement assigns Speaker A to source and temporal modules and Speaker B to filter blocks, testing whether prosody and vocal-tract resonances follow their respective speakers.The A-B condition is compared with A-A and B-B self-reconstruction baselines.Source: C. Explicit Control and… · p. 8
  • Qualitative checks examine articulator contrasts in “pat” and “cat” and the anticipatory retroflex gesture and F3 lowering for American English /r/ in “cart”.The minimal-pair analysis checks whether lips versus tongue dorsum are isolated while shared coarticulatory trends are preserved.Source: C. Explicit Control and… · pp. 8–9
  • Flattening tongue-elevation trajectories in “bird” suppresses acoustic rhoticity, demonstrating direct trajectory-based dialectal control.Audio samples and articulatory-edit examples are available online.Source: C. Explicit Control and… · p. 9, A. AAI Backend Performance · p. 9

A. AAI Backend Performance

  • MMS-FT has the highest seen-speaker correlation at 0.9542, while MMS-SSL leads on unseen speakers at 0.7749; MMS-FT is selected for downstream pseudolabeling.The passage reports that preliminary matched-condition experiments found better downstream TTS performance with MMS-FT than MMS-SSL.Source: A. AAI Backend Performance · p. 9
  • 0.6922 CC is achieved by the proposed TTS model on unseen HPRC speakers, versus 0.7711 for the dedicated MMS-FT AAI baseline on unseen speakers.The TTS model was trained on LibriTTS using MMS-FT pseudolabels, without exposure to HPRC.Source: A. AAI Backend Performance · p. 9

B. TTS Quality and Model Capabilities

The kinematic-conditioned model delivers competitive synthesis quality, with a modest spectral-fidelity cost, and evaluations show controllable prosody and source–filter separation. Cross-speaker tests indicate that pitch follows the source speaker while spectral characteristics follow the filter speaker.

B. TTS Quality and Model Capabilities

  • MMS-FT reaches test-clean WER 5.53 vs 5.96 for Vanilla Source-Filter, MOS 3.76 and UTMOS 3.71, while matching larger XTTS intelligibility (WER 5.53 vs 5.52).The proposed model also outperforms comparably sized YourTTS across perceptual metrics; removing OT-CFM improves intelligibility but degrades naturalness and spectral fidelity.Source: B. TTS Quality and Model Capabilities · pp. 9–10

C. Prosodic and Spectral Fidelity

  • F0 reconstruction reaches CC 0.8841 on test-clean, above XTTS (0.8730) and YourTTS (0.8718), while MCD 5.89 exceeds larger StyleTTS2 (4.52) and HierSpeech++ (4.91).The authors attribute the spectral-fidelity trade-off to the strict source-filter decomposition.Source: C. Prosodic and Spectral Fidelity · p. 10

D. Explicit Source Attributes Controllability

  • Energy CC stays 0.9849–0.9890 across scaling with the lowest Log MSE; StyleTTS2 degrades farther from neutral, while the model exceeds HierSpeech++ on pitch across scales and maintains F1 CC ≈0.88.StyleTTS2 achieves the best raw F0 correlation, and the proposed model's formant tracking remains on par with baselines during source manipulation.Source: D. Explicit Source Attributes… · p. 10

E. Cross-Speaker Disentanglement

  • In cross-speaker synthesis, output F0 aligns more closely with source Speaker A (CC 0.7587) than filter Speaker B (0.6850), while formants and MCD favor Speaker B.A-B MCD is 5.4694 against Speaker B versus 5.8321 against Speaker A, indicating partial but consistent transfer of Speaker B's spectral characteristics.Source: E. Cross-Speaker Disentanglement · p. 11

F. Physical Interpretability and Articulatory Control

The model's predicted articulatory trajectories follow expected physiological patterns and map to corresponding acoustic features. Editing those trajectories can alter a rhotic American token into a non-rhotic variant without acoustic post-processing.

F. Physical Interpretability and Articulatory Control

  • Kinematic grounding provides explicit, interpretable articulatory and accent modification unavailable in opaque TTS systems.The paper presents this as a way to edit dialectal variation in gesture space and probe atypical articulation in physical terms.Source: F. Physical Interpretability and… · p. 11

1) Evaluation of Articulatory Independence in Minimal Pairs:

  • For “pat,” the lower lip rises for bilabial closure; for “cat,” the tongue dorsum elevates for velar closure, while the shared /-æt/ rhyme preserves expected jaw movement.The jaw drops for /æ/ and elevates for final /t/ despite minor spatial variation.Source: 1) Evaluation of Articulatory… · p. 11

2) Acoustic-Kinematic Realisation of the American Rhotic:

  • In “cart,” F3 lowering accompanies tongue-tip elevation and retraction for the retroflex /r/; at final /t/, F3 rises as the tip advances toward alveolar closure.Tongue dorsum and body retraction support the rhotic gesture while the lowered jaw indicates decoupling from the jaw cycle.Source: 2) Acoustic-Kinematic Realisation of… · p. 12

3) Controllability Using Articulatory Trajectory Modification:

  • Flattening tongue elevation in “bird” removes the F3 drop and leaves F3 separated from F2, producing a non-rhotic variant through trajectory editing.The baseline rhotic gesture is marked by an F3 plunge toward F2.Source: 3) Controllability Using Articulatory… · p. 12

VII. CONCLUSION

  • The architecture trades some raw spectral fidelity for structural disentanglement, with source-speaker pitch and filter-speaker spectral characteristics supported by cross-speaker tests.The conclusion also identifies enforcing inter-articulator correlations and scaling toward state-of-the-art audio quality as future work.Source: §VII · p. 12
Loading 2610.00735v1…