PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

Eric Ming Chen, Jin Woo Lee, Vincent Sitzmann

arXiv:2609.36737v1cs.SDcs.CLeess.AS

TL;DR

Recovering vocal-tract shape from speech is an inverse problem for which existing tools lack an efficient differentiable physical simulator. This paper introduces a GPU-accelerated acoustic-tube simulator with gradient-based reconstruction, and reports MRI-video recovery from speech without paired speech-MRI data, outperforming the closest paired-data prior work [43].

  • Problem

    Reconstructing vocal-tract geometry from speech is an inverse problem, while existing vocal-tract simulators are computationally expensive and CPU-bound for deep-learning pipelines.

    Source: §1 · pp. 1–2

  • Method

    A differentiable GPU-accelerated acoustic-tube simulator propagates speech and uses gradients, with neural-network geometry parameterization, to reconstruct vocal tracts from audio.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    The method reconstructs moving vocal-tract MRI video from speech without paired speech-MRI data and outperforms the closest paired-data prior work [43].

    Source: §1 · p. 2

  • Takeaways & Limitations

    The simulator establishes a physically grounded link between vocal-tract geometry and speech and supports self-supervised audio-to-shape learning and MRI-video reconstruction.

    Source: §1 · p. 2

    The inverse problem is ill-posed because different geometries can produce the same sound, and the MRI dataset does not image the nasal tract.

    Source: §7 · p. 10

Abstract

from arXiv · show

The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.

1 Introduction

Learning speech requires solving an inverse mapping from sound to vocal-tract movement, but existing tools do not provide an efficient differentiable simulator for that task. This paper introduces a GPU-accelerated physical simulator and uses it to reconstruct vocal-tract geometry and MRI video from speech.

1 Introduction

  • Existing vocal-tract simulators are computationally expensive and CPU-bound, limiting their use in deep-learning pipelines and speech-to-motor reconstruction.Room-acoustics simulators use ray-based methods ill-suited to the vocal tract’s time-varying morphology, while neural vocoders ignore underlying physics.Source: §1 · pp. 1–2
  • The simulator propagates sound through an acoustic-tube model and uses gradients to infer vocal-tract shape from audio.Its frequency-domain formulation is 70× more GPU-parallelizable than time-domain finite differences, and it adds differentiable turbulence for consonants.Source: §1 · p. 2
  • The method reconstructs moving vocal-tract MRI video from speech without paired speech-MRI training data and outperforms the closest paired-data prior work [43].It connects an MRI generative model to the acoustic simulator to recover MRI video and audio jointly.Source: §1 · p. 2, Figure 1 · p. 2
  • A self-supervised autoencoder maps raw audio to vocal-tract shape, without paired articulatory data.The paper frames this infrastructure as enabling physically grounded speech learning and applications in language, music, and healthcare.Source: §1 · pp. 1–2

2 Related Work

Prior approaches either struggle to differentiate through physical vocal-tract models or rely on data-hungry black-box mappings. This work distinguishes itself through physics-based, time-varying acoustic-tube simulation and neural-field geometry reconstruction.

Articulatory Synthesis

  • Classical vocal-tract simulators, including Kelly-Lochbaum and VocalTractLab, are difficult to backpropagate through and use separate vowel and consonant forward models.Their finite-difference treatment of sound propagation and discontinuous control flow limit direct gradient-based optimization.Source: §2 · p. 2

Black Box Methods

  • Black-box speech-to-articulation networks are hard to interpret physically and depend on sparse paired training data, unlike this method [43] [9].Related examples use VocalTractLab-generated sounds, MRI, or electromagnetic articulography measurements.Source: §2 · pp. 2–3

Differentiable Digital Signal Processing

  • Unlike unconstrained DDSP speech models [15] [61], the physical simulator models both poles and zeros, including radiation and viscous losses.Schulze-Forster et al. use an all-pole vocal-tract filter.Source: §2 · p. 3

Differentiable Acoustic Simulation with Neural Fields

  • Room-acoustic neural fields use ray approximations [37] [60] [34] [35], but vocal-tract wavelengths exceed tract diameter, making wave effects dominant.The simulator instead models time-varying acoustic tubes with explicit physics.Source: §2 · p. 3
  • Südholt et al. [57] reconstruct vocal-tract areas with explicit geometry but demonstrate only static vowels and consonants; this work uses neural fields for moving geometry and full words.The neural-field parameterization is the key difference identified in the comparison.Source: §2 · p. 3

3 Background: The Acoustics of Speech

Speech production is modeled as a source filtered by the vocal tract, represented as a changing one-dimensional acoustic tube. The choice of synthesis domain matters for optimization, with frequency-domain synthesis producing smoother loss landscapes than time-domain synthesis.

3 Background: The Acoustics of Speech

  • Speech is modeled as vocal-fold vibration or turbulent noise filtered by the vocal tract’s acoustic resonance to form phonemes.The resonator’s frequency-response peaks are formants, important auditory cues for vowel perception.Source: §3 · pp. 3–4

Acoustic Tubes

  • The vocal tract is represented as a one-dimensional tube with cross-sectional area A(x), dynamically changed by articulators such as the tongue, mouth, and jaw.Its sound propagation is governed by the linearized Euler equations.Source: §3 · p. 3

Acoustic Tubes as Source-Filter Models

  • For a quasi-static geometry, the tract acts as an LTI filter mapping glottal input u_in to lip-radiation output p_out; its frequency response contains formants.The input is modeled with the Liljencrants–Fant model, and the lips use a radiation boundary condition.Source: §3 · p. 3

Discretization

  • The area function is discretized into uniform spatial and temporal sections with piecewise-constant area, and the simulator maps (u_in, A) to p_out.The frequency-domain formulation yields smoother loss landscapes than time-domain synthesis when interpolating vowel geometries.Source: §3 · p. 4, Figure 2 · p. 4

4 Problem Statement and Method Overview

The method infers the vocal-tract area function that best matches a target speech spectrogram, using gradients through a differentiable simulator. Its design addresses efficient synthesis, consonant modeling, and non-convex reconstruction.

4 Problem Statement and Method Overview

  • Given target speech, the method estimates vocal-fold vibration and optimizes an area function to minimize the spectrogram distance between synthesized and target audio.The output is produced by applying the simulator Φ to the estimated input and candidate geometry.Source: §4 · p. 4
  • Frequency-domain synthesis enables GPU parallelization 70× faster than time-domain synthesis and stable gradient flow.The frequency-domain method addresses both computation and optimization stability.Source: §4 · p. 4
  • A differentiable turbulence model captures consonant-forming nonlinear effects and unifies vowel and consonant synthesis in one forward pass.Turbulence is not captured by the linearized Euler equations.Source: §4 · p. 4
  • Parameterizing the area function with a neural network accelerates convergence on the non-convex inverse problem.The paper reports that coupling the simulator with neural networks also enables applications in linguistics and medical imaging.Source: §4 · p. 4

5 Differentiable Speech Simulation

The section develops frequency-domain synthesis for efficient, stable vocal-tract simulation, then adds differentiable turbulence modeling for consonants. Its comparisons report better acoustic reconstruction with FDS than TDS and substantially faster synthesis.

5.3 Comparing Frequency Domain Synthesis (FDS) to Time Domain Synthesis (TDS)

  • FDS substantially outperforms TDS across SI-SDR, STOI, and PESQ, with TDS nearly 10 dB lower in SI-SDR across configurations.FDS is also robust to window and hop length; the selected setting is a 25 ms window with a 1/4 hop.Source: Table 1 · p. 5, §5.3 · p. 6
  • FDS averages 70x faster than TDS on GPU; for a one-second utterance, synthesis takes 14.2 ms versus 1.08 seconds.The runtime comparison uses an RTX 4090, 25 ms windows, and a 1/4 hop.Source: Figure 3 · p. 5, §5.3 · p. 6

5.1 Frequency Domain Synthesis (FDS)

  • FDS converts the governing equations into a frequency-domain spatial solve, avoiding temporal recursion and allowing independent parallel solutions at each frequency.A chain-matrix product across sections yields the tract filter; the output spectrum is recovered as speech with an inverse Fourier transform.Source: §5.1 · p. 5
  • For time-varying tracts, the method synthesizes successive windows independently and combines them with overlap-add.It applies a Tukey analysis window to the input and a Hann synthesis window to the output.Source: §5.1 · p. 5

5.2 Differentiable Gating of Consonants

  • To model consonants, the method treats airflow through narrow constrictions as turbulence, whose Reynolds number rises as the area narrows.It replaces discontinuous constriction selection and gating with softmax weighting and a softplus function so gradients can flow through the turbulence model.Source: §5.2 · pp. 5–6
  • The unified harmonic-plus-noise formulation supports dynamic vowel-to-consonant transitions while preserving differentiability.Although turbulence is computed in the time domain, the technique runs 70x faster than real time.Source: §5.2 · p. 6

5.3 Comparing Frequency Domain Synthesis (FDS) to Time Domain Synthesis (TDS)

  • Neural fields produce smoother tube reconstructions than directly optimized discrete areas, which can remain near initialization or develop high-frequency oscillations.The figure attributes the smoothness to shared weights coupling the geometry of each section.Source: Figure 4 · p. 7

6 Experiments

The experiments evaluate neural-field reconstruction, self-supervised speech autoencoding, and MRI-video reconstruction through a differentiable acoustic simulator. They report improved reconstruction and intelligibility outcomes, alongside phonetic structure emerging without supervision.

6.1 Neural Networks Escape Sub-optimality

  • Both neural fields significantly outperform the discrete baseline; shared weights support coarse-to-fine fitting where discrete optimization stalls.The neural-field reconstructions recover general area-function morphology, with formants aligning accurately to ground truth.Source: §6.1 · pp. 7–8, Table 2 · p. 8

6.2 Self-supervised Autoencoding of Speech

  • Across 11 languages, the self-supervised autoencoder produces more intelligible utterances and better preserves speaker identity than TensorTract2.The evaluation reports WER, CER, and identity metrics with 95% confidence intervals.Source: §6.2 · p. 8, Table 3 · p. 9
  • The 32-section area functions form a continuous phoneme spectrum ordered by place of articulation, from vowels to consonant constrictions.The analysis uses a UMAP projection of encoded area functions.Source: §6.2 · p. 8, Figure 5 · p. 8

6.3 Physically-Grounded MRI Visualization from Speech

  • MRI reconstruction optimizes per-frame StyleGAN2 latents end-to-end through extracted tract areas and the acoustic simulator, without paired speech–MRI data.Cross-sectional areas are extracted from each generated frame and used to synthesize the target utterance.Source: §6.3 · pp. 8–9
  • Without phonetic labels, formants from 1,000 GAN-generated MRI frames form a convex hull resembling the IPA vowel chart, with closely corresponding tongue positions.The authors interpret this as evidence that the simulator captures geometrically meaningful vocal-tract configurations.Source: §6.3 · p. 9, Figure 6 · p. 9
  • FVD is 2949 for Speech2rtMRI versus 623 for our method; ours also outperforms it on SSIM and LPIPS and produces intelligible audio.The comparison evaluates visual and acoustic quality.Source: §6.3 · p. 10

7 Conclusion

The conclusion summarizes a differentiable, GPU-parallelizable physics-based speech model and two applications that do not require paired articulatory or speech–MRI data. It also notes that inverse ambiguity and the MRI dataset’s coverage constrain reconstruction.

7 Conclusion

  • The work combines frequency-domain acoustics, a differentiable consonant-formation model, and neural-field area functions to support two unpaired-data applications.The applications are self-supervised audio-to-area-function autoencoding and speech-to-MRI-video visualization.Source: §7 · p. 10
  • Different geometries can produce the same sound, and the MRI dataset omits the nasal tract, limiting nasal-tract reconstruction.The authors note that including the nasal tract could produce more faithful formants and anti-formants.Source: §7 · p. 10
  • Potential applications include voice coaching or singing synthesis, language-acquisition research, and speech pathology.The paper also identifies extending the model to other animals as future work.Source: §7 · p. 10

A.1 Derivation of the Linearized Euler Equations

The appendix derives tract acoustics from linearized fluid equations and represents the tract as a frequency-domain transmission line. It also describes transfer functions, nasal-tract modeling, and a time-domain formulation.

A.1 Derivation of the Linearized Euler Equations

  • Linearization assumes small velocity, density, and pressure perturbations, drops second-order terms, and therefore cannot synthesize turbulent flow.For frequencies below 4 kHz, the tract is modeled as dominated by plane waves, enabling a one-dimensional formulation.Source: §A.1 · p. 16

A.2 Circuit Interpretation and the Transmission Line Model

  • Viscous wall friction adds a frequency-dependent resistance, while yielding walls are modeled as damped oscillators that add shunt impedance.The ideal-fluid circuit model has purely imaginary impedances; wall friction and wall motion add losses and compliance effects.Source: §A.2 · pp. 17–18

A.3 Transfer Functions

  • Concatenated section chain matrices provide pressure and volume-velocity transfer functions between the glottis, tract sections, and lips.The transfer-function derivation applies glottal and lip boundary conditions and also identifies backward impedance from a section toward the glottis.Source: §A.3 · pp. 18–19

A.4 The Nasal Tract

  • Nasal geometry is not reconstructed because it is absent from the MRI frames, but its acoustic effect can be represented by a bifurcated transmission line or lumped equivalent area function.The nasal and oral systems can be combined as an equivalent circuit consisting of a single transmission line.Source: §A.4 · p. 20

B.1 Time Domain Governing Equation

  • The time-domain model replaces frequency-dependent wall resistance with DC resistance and expresses the circuit as coupled ordinary differential equations.The lip radiation load adds two flow unknowns, and radiated pressure is the voltage across the lip resistance.Source: §B.1 · p. 20

B.2 Numerical Scheme

The time-domain baseline uses a semi-implicit scheme, while the paper’s inverse problem estimates a time-varying area function directly from real speech without requiring ground-truth geometry.

B.2 Numerical Scheme

  • With ϑ = 0.5, the time-domain equations reduce to a tridiagonal linear system solved by recursion and a tridiagonal solver; this simulation serves as a validation baseline.The selected value corresponds to the trapezoidal scheme.Source: §B.2 · p. 20
  • Gradient-based optimization estimates A(x, t) from real-world speech ptgt without presupposing a ground-truth area function.The paper describes several possible gradient-based optimization techniques for this inverse problem.Source: Figure 8 · p. 21

C Experimental Details

The experiments use a fixed spatial discretization and generally run at 16 kHz, with higher sample rates used for singing examples. Speech fitting uses a multiscale log-Mel spectrogram loss.

C.1 Simulation Details

  • The simulator uses 32 spatial segments and generally runs at 16 kHz, while singing examples use sample rates from 32 to 44.1 kHz.The simulation is modeled in centimeters; higher fidelity is possible at higher sample rates.Source: §C.1 · p. 21

C.2 Loss Function

  • The target and simulated pressures are compared using averaged L2 distances between log-Mel spectrograms at three resolutions.The configurations use FFT sizes 512, 1024, and 2048, with corresponding window lengths 160, 400, and 800 and hop lengths 40, 80, and 160.Source: §C.2 · p. 21

C.3 Evaluation Setup for Perceptual Speech Quality Metrics

The evaluation fits 25 LibriTTS-R speech samples and assesses perceptual quality with SI-SDR, STOI, and PESQ. For real-world speech reconstruction, both tested neural fields outperform the discrete representation across all reported speech metrics.

C.3 Evaluation Setup for Perceptual Speech Quality Metrics

  • The perceptual-quality evaluation fits 25 random LibriTTS-R speech samples using an MFN area function, Adam, and 200 optimization steps.It reports SI-SDR, STOI, and PESQ using TorchAudio-Squim.Source: §C.3 · p. 21

C.4 Neural Fields

  • The compared area-function parameterizations are a four-layer RFF network and a three-layer MFN, each trained for 200 steps.Both use Adam with a learning rate of 10^-2; fitting a five-second sample takes approximately one minute.Source: §C.4 · p. 22
  • Both neural fields significantly outperform the discrete representation across all speech metrics on 25 reconstructed LibriTTS-R samples.The evaluation uses the experimental setup described for perceptual speech quality metrics.Source: §C.4 · p. 22

C.5 Autoencoding

The appendix describes the autoencoder training and phoneme-space projections, then details MRI reconstruction and its diffusion-model comparison, before outlining responsible-use guidance.

C.5 Autoencoding

  • The autoencoder uses a Wav2Vec 2.0-based CNN and 12-layer transformer, trained on English audio before language-specific fine-tuning.Training uses 3-second segments, batch size 64, and 15,000 iterations; it is fine-tuned for 10 epochs per other language.Source: §C.5 · p. 22
  • Training data comprise LibriTTS-R for English, Multilingual LibriSpeech for other European languages, and KSS, JVS, and AISHELL-3 for Korean, Japanese, and Chinese.Source: §C.5 · p. 22

C.5.1 UMAP Projection

  • Encoded area functions remain disentangled by phoneme while forming a continuous spectrum across a larger set of IPA phonemes.The accompanying UMAP projection visualizes this continuous space.Source: §C.5.1 · p. 22, Figure 9 · p. 23

C.6 MRI Generation

  • MRI reconstruction trains a StyleGAN2 model conditioned on dataset identity labels and uses 30 latent vectors to reconstruct 3-second clips.MRI audio is enhanced to reduce noise from the scanner's spinning magnets before reconstruction.Source: §C.6 · p. 22

C.6.1 MRI Reconstruction with Diffusion Models

  • Diffusion Posterior Sampling performed worse than the GAN and could not make large geometric changes such as tongue movement.The authors describe blind inverse problems with diffusion models as an open problem.Source: §C.6.1 · p. 22

D Ethical Discussion

  • The authors state that the simulator targets realistic speech reconstruction rather than speech generation and call for clear labeling of its samples.They also emphasize responsible use amid ethical concerns around generative speech models and tasks such as voice cloning.Source: D Ethical Discussion · p. 23

Related papers

Loading 2609.36737v1…