PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous-Time Acoustic Modelling with Neural Controlled Differential Equations

Mattias Cross, Minghui Zhao, Anton Ragni

arXiv:2609.11725v1cs.SDcs.AI

TL;DR

TTS length regulation aligns text and acoustic frames by repeating phone-level states, but does not let duration change their values. This paper uses neural CDEs to evolve hidden states along a duration-aware phonetic path. Results indicate small benefits on neutral audiobook-style speech and improved reference-style tracking in emotional TTS, with comparable utterance-level emotion-expression quality.

  • Problem

    Length regulation aligns text and acoustic frames by repeating phone-level states, but repeated states do not explicitly represent position-dependent temporal evolution within phone segments.

    Source: §I · p. 1, C. Continuous-Time Modelling with CDEs · p. 2

  • Method

    A neural CDE evolves hidden states in response to a phone control path and duration-derived time, providing a continuous-time bridge between text and acoustic representations.

    Source: C. Continuous-Time Modelling with CDEs · p. 3, B. Training a CDE within Matcha-TTS · p. 4

  • Results

    CDEs provide a small benefit on neutral audiobook-style speech and improve reference-style tracking in emotional TTS, with a single-layer CDE at h = 1.0 achieving the strongest macro Spearman correlation and comparable utterance-level emotion-expression quality.

    Source: §IV · pp. 5–6, §V · p. 6

  • Takeaways & Limitations

    The results position CDEs as a continuous-time, duration-aware design option for style-sensitive TTS.

    Source: §V · p. 6

Abstract

from arXiv · show

Text-to-speech (TTS) models commonly address text--speech alignment by expanding phone-level encoder states to frame-level decoder inputs using predicted durations. While this length-regulation step resolves alignment structurally, this use of duration typically changes only where and how often latent states appear, not the values of the states themselves. This paper proposes a continuous-time mechanism for duration-aware acoustic modelling in TTS using neural controlled differential equations (CDEs). We formulate the phone representation as a temporally parameterised control path and use a neural acoustic vector field to produce a continuous-time hidden state whose values evolve with phonetic content and duration-derived timing. The resulting trajectory can be sampled at discrete points and integrated into a standard acoustic decoder pipeline. Objective results contrast CDEs and typical recurrent models. Subjective results suggest that CDE-based models evaluating one phone per step can improve rank-order agreement between synthesised and reference emotion intensity while maintaining comparable emotion-expression quality to a strong baseline. Additional experiments with half-phone step-sizes suggest that temporal resolution changes the trade-off between style tracking and absolute calibration. These results position CDEs as a promising design space for continuous-time and duration-aware style-sensitive TTS.

I. INTRODUCTION

The introduction frames TTS alignment as a duration-varying mapping from text units to acoustic units. It proposes neural CDEs to let duration-derived timing shape latent values over a continuous phonetic trajectory.

I. INTRODUCTION

  • TTS serves speech-centered technologies and must produce intelligible, natural, temporally structured speech adaptable to speakers and styles.Source: §I · p. 1
  • Phones vary in duration across utterances, contexts, and speakers, making text-to-acoustic alignment irregular..Source: §I · p. 1
  • Without a CDE, the figure shows Htxt repeated according to predicted durations without those durations changing decoder-input values.Source: Figure 1 · p. 1
  • Conventional length regulation changes where and how often latent states appear, but leaves their values unchanged across repeated frames.The introduction notes that a lengthened vowel may instead evolve in pitch, energy, spectral tilt, and coarticulation.Source: §I · p. 1
  • CDE-based TTS uses a neural acoustic vector field to evolve hidden states with phonetic content and duration-derived timing.The resulting trajectory can be evaluated at phone boundaries or intermediate points.Source: §I · p. 2
  • One-phone-per-step CDE evaluation produced significant Spearman rank correlation between synthesised and reference emotion intensity while maintaining the same emotion-expression quality.The introduction presents this as subjective-evaluation evidence for style-sensitive TTS.Source: §I · p. 2

A. Encoder–Decoder Architecture

The encoder–decoder architecture maps phone labels to phone-level representations, then repeats them to match acoustic-frame duration. Style embeddings condition prosody and decoding, but length-regulated acoustic states remain unchanged by duration.

A. Encoder–Decoder Architecture

  • The text encoder maps phone labels to C-dimensional phone-level representations, which are repeated according to predicted or reference durations to match acoustic-frame length.A duration predictor supplies alignments at test time, and the resulting acoustic features feed a decoder or decoder–vocoder.Source: A. Encoder–Decoder Architecture · p. 2

B. Style Embeddings

  • Style embeddings condition prosodic features, style, pitch, energy, and duration, while length-regulated acoustic embeddings remain invariant in value to duration.The style embedding is predicted from a reference audio sample using prosodic text representations.Source: B. Style Embeddings · p. 2

C. Continuous-Time Modelling with CDEs

CDEs address the limits of length regulation by making hidden states evolve continuously with phone content and duration-derived timing. The resulting trajectory can be sampled for decoder use, with step size controlling temporal resolution and computational cost.

C. Continuous-Time Modelling with CDEs

  • Length regulation repeats unchanged encoder states, limiting access to local speaking rate, phone-boundary proximity, and temporal progression.Frames may have similar values despite coming from phones with different durations or positions within a phone.Source: C. Continuous-Time Modelling with CDEs · p. 2
  • Phone embeddings are augmented with a time channel derived from cumulative phone durations, and the discrete controls are interpolated into a continuous path.This work uses linear interpolation and leaves other schemes for future work.Source: C. Continuous-Time Modelling with CDEs · pp. 2–3
  • A neural vector field evolves the hidden state in response to changes in both phone conditioning and the duration-derived time channel.The trajectory can be sampled at frame level directly or at phone level and aligned for decoder compatibility.Source: C. Continuous-Time Modelling with CDEs · p. 3
  • Step size h sets solver resolution: h = 1 processes one phone per step, while h = 0.5 adds an intermediate evaluation and increases function evaluations.Timing remains encoded in the control path even when evaluations occur at regular intervals.Source: C. Continuous-Time Modelling with CDEs · p. 3
  • The continuous hidden trajectory can serve as a control for a subsequent CDE layer, providing a design space for stacked continuous conditioning.The adjoint state method can reduce memory by avoiding storage of the full forward trajectory.Source: C. Continuous-Time Modelling with CDEs · p. 3

D. Efficient Stacking of CDE Layers

Stacked CDE layers can be propagated jointly over the same control path, avoiding separate solves and intermediate trajectory reconstruction. This efficiency requires the layers to share a numerical solver and integration grid.

D. Efficient Stacking of CDE Layers

  • The construction expresses the stacked dynamics directly with respect to the original control path through the chain rule for CDEs.Source: D. Efficient Stacking of CDE Layers · p. 3
  • A joint augmented solve over the same control avoids a separate first-layer solve, interpolation of its sampled states, and a second solve.The lower layer represents the aligned phone-duration trajectory, while the upper layer integrates it into a higher-level conditioning state.Source: D. Efficient Stacking of CDE Layers · p. 3
  • The stacked layers must share the same numerical solver and integration grid; different solvers, tolerances, or output resolutions favor a more flexible two-stage interpolation approach.Source: D. Efficient Stacking of CDE Layers · p. 3

A. Data

The experiments use neutral audiobook narration and emotional speech, with the emotional data providing reference utterances for style prediction. The emotional subset and evaluation splits are limited in size.

A. Data

  • LJSpeech is a 24-hour audiobook-style narration dataset spoken by one U.S. speaker, and its prosody is described as neutral.Experiments use the same split as.Source: A. Data · p. 3
  • The English ESD subset contains 10 hours from 10 speakers across five emotions, with 30 minutes per speaker selected for fine-tuning.The held-out validation and test sets each contain 100 samples; subjective testing uses 50 test samples.Source: A. Data · pp. 3–4
  • ESD evaluation samples have same-speaker, same-emotion reference utterances, while the paper describes the five-hour subset as challenging for learning nuanced prosody from limited data.The challenge includes subtle energy and pitch cues that are only partially captured by TTS training losses.Source: A. Data · p. 4

B. Training a CDE within Matcha-TTS

The Matcha-TTS experiment inserts a trainable CDE between a pretrained encoder and decoder to test whether the CDE can bridge their text and acoustic representations.

B. Training a CDE within Matcha-TTS

  • Only the CDE is trainable, using the Matcha encoder as its control path and the decoder as the Mel-spectrogram readout.This tests insertion into a pretrained TTS pipeline without disrupting its encoder-decoder mapping.Source: B. Training a CDE within Matcha-TTS · p. 4

C. Emotional TTS Guided by Reference Style

The system uses a CDE to transform an encoder-produced control path into acoustic representations for a decoder, targeting speech that matches a reference style and emotion intensity. A duration-augmented control path is also used to compare CDE configurations with an LSTM.

C. Emotional TTS Guided by Reference Style

D. Implementation Details

The implementation uses a U-Net parameterization for the CDE vector field and integrates its sampled states into the acoustic model. The section also describes subjective and objective evaluation protocols for emotion expression and acoustic differences.

D. Implementation Details

  • A U-Net parameterizes the CDE vector field to avoid the prohibitive final-layer expansion of an MLP, and the CDE is integrated with fixed-step fourth-order Runge–Kutta.The sampled hidden sequence is upsampled and added to text states through a trainable gated residual connection.Source: D. Implementation Details · p. 4
  • The initial CDE state is predicted by a reverse-sequence LSTM, while the control-path time channel is normalized by total phone duration.Source: D. Implementation Details · p. 4

E. Evaluation

  • MOS-EI uses a 1–5 scale to rate emotion intensity, and score correlation with the reference indicates calibration to its intensity.Twenty-five participants were recruited, with each sample heard by an average of ten listeners.Source: E. Evaluation · pp. 4–5
  • Table I concerns the effect of training a CDE bridge between a pretrained encoder and decoder on LJSpeech.Source: Table I · p. 5
  • CMOS-EQ compares a candidate against a designated baseline on a +3/−3 scale for relative expression of the target emotion.Source: E. Evaluation · p. 5
  • Emotion accuracy is assessed through participant classification, while MCD and log-F0 RMSE measure spectral difference and pitch-contour error, respectively.Lower MCD correlates with audio quality or similarity; lower log-F0 RMSE correlates with correct intonation, emotion, and prosody.Source: E. Evaluation · p. 5

IV. RESULTS AND DISCUSSION

CDE configurations trade off emotion-category reference-style tracking and calibration: the single-layer, h = 1.0 model reaches a macro correlation of 0.53 versus 0.08 for the ESD fine-tuned baseline. Emotion-expression quality is comparable to the baseline at the utterance level, while listener-level preference and other outcomes vary by emotion and configuration.

IV. RESULTS AND DISCUSSION

  • On LJSpeech, CDE objective scores were comparable to the pretrained mapping, with small numerical differences favoring CDE.The CDE did not disrupt the encoder–decoder mapping.Source: §IV · p. 5
  • CDE and LSTM configurations showed different objective behavior on the same duration-augmented inputs; longer context benefited all models, while F5 suffered pacing inaccuracies.The results indicate that CDE is not a direct replacement for the discrete recurrent baseline.Source: Table II · p. 5, §IV · p. 5
  • 0.53 macro correlation for the single-layer h = 1.0 CDE exceeds 0.08 for the ESD fine-tuned baseline.This configuration hinders neutral speech, improves calibration in other emotions, and performs worse for surprise; changing step size and layer count has non-monotonic effects.Source: §IV · p. 5, Table III · p. 6
  • Per-sample emotion-expression ratings were statistically indistinguishable, while per-listener ratings showed a significant (p < 0.05) preference for CDE.The comparison used a one-layer CDE with h = 1.0; CDE remained significantly farther from ground truth.Source: Table IV · p. 6, §IV · p. 6
  • Human emotion accuracy moderately favored the baseline overall; the best reported result among the other systems was 0.558 for two-layer half-step CDE on sadness.The authors suggest selecting among multiple model configurations for the intended style.Source: Table V · p. 6, §IV · p. 6

V. CONCLUSION

The conclusion presents neural CDEs as a continuous-time, duration-aware approach to acoustic modelling in TTS. Results point to improved emotion-style tracking, comparable utterance-level expression quality, and trade-offs shaped by temporal resolution and model depth.

V. CONCLUSION

  • A single-layer CDE with h = 1.0 achieved the strongest macro Spearman correlation with reference emotion-intensity ratings, with comparable utterance-level quality and modest listener-level preference.The conclusion also reports a small benefit on neutral audiobook-style speech and improved reference-style tracking in emotional TTS.Source: §V · p. 6
  • CDEs use duration-derived timing and a neural acoustic vector field to evolve phone-level hidden states with phonetic content and temporal structure.This contrasts with using duration only to determine where and how often latent states appear.Source: §V · p. 6
  • Step size and layer depth affect the trade-off between style tracking, calibration, and perceived emotion quality, making temporal resolution a meaningful modelling choice.Source: §V · p. 6
  • Future work is proposed on interpolation schemes, solver choices, frame-level control paths, and broader evaluation across speakers, styles, and data resources.Source: §V · p. 6
Loading 2609.11725v1…