Source-linked AI summary
Anatomy-aware cross-speaker adaptation of complete vocal-tract acoustic-to-articulatory inversion
Nhat-Nam Nguyen, Pierre-Andre Vuissoz, Yves Laprie
TL;DR
Cross-speaker acoustic-to-articulatory inversion must account for anatomical and geometric differences, while paired recordings are costly. The paper adapts a fixed model’s contour predictions using anatomical landmarks and affine-TPS mapping; Affine12+TPS14 achieves the lowest Full10 error, 3.19 mm. The findings support the combined value of anatomical correspondences and nonlinear alignment.
Problem
Anatomical and articulatory variation makes inversion generalization to unseen speakers challenging, and paired acoustic-articulatory recordings are costly to collect.
Source: §1 · p. 1Method
The framework uses single-frame anatomical landmarks to map a fixed inversion model’s complete contour predictions to unseen speaker geometries without retraining.
Source: §1 · p. 1Results
3.19 mm Full10 error with A12+T14, lower than A14’s 3.69 mm, with a significant reduction relative to A14 (p = 0.025).
Source: §4 · p. 3Takeaways & Limitations
A12+T14 performs best overall, suggesting that additional lower-incisor correspondences help constrain nonrigid adaptation.
Source: §5 · p. 4A fixed mapping may only partially capture speaker-specific articulation and pose differences, and calibration-frame and landmark-annotation sensitivity remains unquantified.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Cross-speaker acoustic-to-articulatory inversion requires accounting for anatomical differences between speakers. We propose a geometric adaptation framework that uses anatomical landmarks, primarily on vertebrae and dental structures,to transfer predictions from a fixed inversion model to unseen speakers. An affine transformation followed by thin-plate spline (TPS) deformation maps the predicted contours of 10 vocal-tract structures into each target speaker's geometry without retraining. Landmarks are identified in one selected /u/ frame per speaker as a common phonetic reference without assuming identical articulatory configurations across speakers, and the resulting mapping is reused across recordings. We train the model on a single-speaker rt-MRI database and evaluate adaptation on eight speakers from a separate multi-speaker rt-MRI database. We compare affine and TPS configurations using 12 or 14 landmarks. Affine12+TPS14 achieves the lowest mean point-to-closest-point error of 3.19mm. These results support the combined value of anatomical landmark information and nonrigid alignment.
1. INTRODUCTION
Cross-speaker inversion is challenged by anatomical and articulatory variation, while transferring complete vocal-tract contours also requires addressing geometric mismatch. The paper proposes single-frame anatomical calibration to adapt a fixed inversion model to unseen speaker geometries.
1. INTRODUCTION
- Anatomical and articulatory variation across speakers, together with costly paired recordings, complicates generalization of acoustic-to-articulatory inversion.Speaker-independent models and adaptation methods address this challenge using vocal tract length normalization and self-supervised speech representations.Source: §1 · p. 1
- rt-MRI captures pharyngeal and laryngeal anatomy, extending beyond the sparse sensors tracked by EMA.Prior work reconstructed rt-MRI images from speech and enabled contour-based representations; Azzouz et al. demonstrated acoustic reconstruction of contours spanning the vocal tract from glottis to lips.Source: §1 · p. 1
- Transferring complete vocal-tract contour predictions to unseen speakers requires addressing geometric mismatch.Prior registration work includes affine mappings for EMA and TPS warping of multi-speaker EMA data to a shared template.Source: §1 · p. 1
- Anatomical landmarks transfer complete MRI-derived contour predictions from a fixed inversion model to unseen speaker geometries using single-frame calibration, without retraining.The study examines landmark coverage and nonlinear alignment under this constraint.Source: §1 · p. 1
- The framework evaluates affine and affine-TPS configurations with 12 or 14 landmarks across 10 structures, training on one speaker and testing on eight others.The tested structures include lips, tongue, soft-palate midline, pharyngeal wall, epiglottis, arytenoid cartilage, vocal folds, and incisors.Source: §1 · p. 1
2. DATASETS
The study trains its reference inversion model on ASD2 and evaluates cross-speaker adaptation using ASD1 recordings from other speakers.
2. DATASETS
- ASD1 contains five male and five female native French speakers, each contributing about 15 minutes of speech across 77 sentences and 16 sessions.P4 has 15 usable sessions.Source: §2 · pp. 1–2
- Evaluation uses approximately 130,000 non-silent ASD1 frames from P1 and P3–P9, excluding P2 because laryngeal visibility is insufficient.The evaluated recordings span eight speakers.Source: §2 · p. 2
- ASD2 supplies approximately 3.5 hours of speech from one native French female speaker for training the reference model.The speaker also appears as P10 in ASD1, providing a shared identity across corpora.Source: §2 · p. 2
- Each of the 10 vocal-tract structures is represented by 50 two-dimensional points, yielding 1,000 output coordinates per frame.Source: §2 · p. 2
3. METHODS
The methods combine a fixed acoustic-to-articulatory model with anatomical calibration and geometric mapping, then evaluate contour reconstruction across speakers and adaptation configurations.
3.1. Acoustic-to-articulatory inversion model
- The reference inversion model uses two fully connected layers and two bidirectional LSTM layers, with global coordinate-wise normalization statistics computed from reference training data.Those same statistics are reused at validation and inference to recover reference-space contour coordinates; target-speaker contour sequences are not needed for normalization.Source: §3.1 · p. 2
3.2. Geometric adaptation
- Each target speaker receives a fixed reference-to-target mapping estimated from anatomical landmark correspondences and reused across recordings.Source: §3.2 · p. 2
3.2.1. Anatomical landmarks
- The 12-landmark set uses upper-incisor and hard-palate points, cervical vertebra centers, and a pharyngeal-wall landmark; the 14-landmark set adds two lower-incisor extrema.The additional landmarks extend coverage to the lower anterior oral cavity and constrain mandibular alignment.Source: §3.2.1 · p. 2, Figure 1 · p. 2
3.2.2. Geometric transformation
- The proposed mapping fits an affine alignment using 12 landmarks, then uses 14 affine-transformed landmarks to constrain a nonrigid TPS deformation applied pointwise to predicted contours.TPS uses exact interpolation with zero smoothing and a first-degree polynomial term.Source: §3.2.2 · pp. 2–3
3.3. Evaluation
- P2CPmean measures contour reconstruction without requiring matching point indices, and aggregate means give equal weight to the selected structures.Errors are averaged over frames within sessions, then equally across sessions and speakers; standard deviations describe pooled frame-level errors.Source: §3.3 · p. 3
3.4. Experimental setup
- Experiments keep the ASD2-trained model fixed and calibrate each target speaker with one /u/ frame, reusing the resulting mapping across recordings.Configurations compare affine and affine-TPS mappings with 12 or 14 landmarks; the /u/ reference is selected without assuming identical articulation across speakers, and A12+T14 is compared with A14 using paired tests.Source: §3.4 · p. 3
4. RESULTS
Affine12+TPS14 achieves the lowest Full10 error among the tested configurations, while structure-level and speaker-level results show that benefits are not uniform.
4. RESULTS
- The ASD2-trained model has mean P2CPmean of 1.40 mm across 10 structures, while unadapted target-speaker Full10 error reaches 8.87 mm.Table 1 reports the ASD2 model's P2CPmean in millimeters; Table 3 presents speaker-wise Full10 P2CPmean.Source: §4 · p. 3
- A12+T14 has the lowest error for six of 10 structures and improves over A14 for eight, but epiglottis and soft-palate errors rise slightly.Only arytenoid cartilage and lower incisor reductions are statistically significant (p < 0.05).Source: §4 · p. 3
- All adaptation configurations outperform Raw for every target speaker; A12+T14 is best for seven of eight, but P4 worsens from 2.95 to 3.17 mm.The seven improvements over A14 are significant (p < 0.05).Source: §4 · p. 3
- 3.19 mm Full10 error for A12+T14 is lower than A12 (4.10 mm), A14 (3.69 mm), and A12+T12 (3.75 mm).The reduction versus A14 is approximately 13.6% and statistically significant (p = 0.025).Source: §4 · p. 3, Table 2 · p. 4
5. DISCUSSION
The comparisons distinguish landmark coverage from nonlinear alignment, while the discussion notes residual anatomical and recording-related limitations and nonuniform gains.
5. DISCUSSION
- Adding landmarks and TPS is associated with better aggregate performance; the authors suggest lower-incisor correspondences help constrain nonrigid adaptation.The comparisons include A14 versus A12 and A12+T12 with the landmark set unchanged.Source: §5 · p. 4
- Residual laryngeal errors may reflect gender-related oral-pharyngeal proportions or laryngeal position, whose contributions are not isolated; pose differences and calibration sensitivity also remain concerns.The mapping may only partially capture speaker-specific articulation and pose differences across unregistered ASD1 recordings; calibration-frame and landmark-annotation sensitivity remains unquantified.Source: §5 · p. 4
- Table 3 reports speaker-wise Full10 P2CPmean (mm) for A14, the main affine baseline, and A12+T14, proposed; marked differences use paired two-sided t-tests on session means within speakers, unadjusted p < 0.05.Source: Table 3 · p. 4, §5 · p. 4
- The P8 /k/ example contrasts orange predictions with cyan ground truth and displays frame-level P2CP errors below each panel.It illustrates use of the fixed mapping beyond the /u/ calibration vowel.Source: Figure 2 · p. 4
6. CONCLUSION
Single-frame anatomical adaptation transfers contours from a fixed AAI model to unseen speakers without retraining. Across eight speakers, Affine12+TPS14 lowers Full10 error relative to Affine14, while future work will extend training and assess robustness and residual errors.
6. CONCLUSION
- Single-frame anatomical adaptation transfers vocal-tract contours from a fixed AAI model to unseen speakers without retraining.The framework is intended for target-speaker adaptation using anatomical information.Source: §6 · p. 4
- Across eight speakers, Affine12+TPS14 reduces Full10 error from 3.69 mm with Affine14 to 3.19 mm.The conclusion supports the combined value of landmark coverage and nonlinear alignment.Source: §6 · p. 4
- Future work will extend the framework to multi-speaker training toward speaker-independent AAI and assess calibration robustness and residual laryngeal and lip errors.Source: §6 · p. 4