PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

Hyung Kyu Kim, Byungchan Hwang, Hak Gu Kim

arXiv:2609.30517v1eess.AScs.GRcs.LG

TL;DR

Speech-driven facial animation remains challenging because acoustics can correspond to multiple motions and existing methods often do not explicitly model visible articulation. The paper models speech through directional articulatory motions and composes them into facial motion; it reports improved speech consistency and preference for lip sync and realism in a user study.

  • Problem

    Speech-to-face motion is one-to-many, and many methods focus on holistic vertex or mesh reconstruction without explicitly modeling structured visible articulation.

    Source: §1 · p. 1

  • Method

    The framework uses SAM to model speech-to-directional-motion correspondence and TAC to compose those motions into surface-consistent facial motion under mesh topology.

    Source: §1 · p. 2

  • Results

    Participants preferred the method across competitors, with average preference rates of 70.3% for Lip Sync and 70.7% for Realism.

    Source: §4.7 · p. 9

  • Takeaways & Limitations

    The authors conclude that structured articulatory directions improve speech consistency and move facial animation beyond holistic vertex regression toward physically grounded, interpretable animation.

    Source: §5 · p. 9

    The method uses FLAME and BIWI topologies without teeth or tongue and does not explicitly model non-articulatory upper-face dynamics such as blinking.

    Source: §A.6 · p. 27

Abstract

from arXiv · show

Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.

1 Introduction

Speech-driven facial animation faces an inherently one-to-many mapping from acoustics to motion, while many methods regress holistic face trajectories without explicitly modeling articulation. The paper proposes modeling visible speech through directional articulatory motions and composing them into facial motion.

1 Introduction

  • Acoustics can map to multiple facial motions because articulators are coupled and compensate under structured constraints, while holistic vertex regression leaves articulation weakly modeled.The paper identifies limited interpretability and physical grounding as shortcomings of holistic regression.Source: §1 · p. 1
  • Visible articulation is represented by spreading–rounding, opening–closing, and protrusion–retraction motions defined from lip and jaw mesh anchors.The motions follow consistent, directionally dominant patterns across representative phonemes.Source: §1 · p. 2
  • The framework uses Speech–Articulatory Memory (SAM) to model speech-to-motion correspondence under phonetic context and Topology-aware Articulatory Composition (TAC) to produce surface-consistent facial motion.SAM uses memory-based retrieval and decoding; TAC composes directional motions under mesh topology.Source: §1 · p. 2
  • The authors report that their articulation-aware framework outperforms state-of-the-art models and improves interpretability through quantitative and qualitative evaluation, including user studies.The contribution passage does not provide metric values.Source: §1 · p. 2

2 Related Works

Prior speech-driven animation methods improve fidelity, topology handling, or controllability, but generally do not explicitly model visible articulatory structure. The paper motivates direct vertex-level modeling of context-dependent articulation rather than combinations of fixed viseme shapes.

2.1 Speech-Driven 3D Facial Animation

  • Speech-driven animation spans temporal CNNs, Transformers, motion and key-motion priors, lip-reading supervision, heterogeneous annotations, and methods for unregistered scans or richer style and emotion.These approaches primarily target motion fidelity, topology handling, or controllability rather than explicit visible articulatory structure.Source: §2.1 · p. 3

2.2 Articulatory Modeling for Speech and Animation

  • Prior articulatory work describes lip opening, spreading or rounding, and protrusion or retraction, while parameterized systems such as JALI and VisemeNet [70] use lip–jaw parameters and fixed viseme shapes.In these systems, phonemes map to predefined visemes that are blended, with context-dependent variation expressed through combinations of those patterns.Source: §2.2 · p. 3
  • Natural articulatory motion varies continuously with phonetic context rather than as combinations of fixed shapes, motivating direct vertex-level modeling.The paper represents articulation with three canonical directions and predicts speech-driven direction-specific motion across the facial surface.Source: §2.2 · p. 3

3 Proposed Method

The method represents facial motion through three speech-grounded directions, retrieves context-sensitive directional motion with SAM, and uses TAC to compose topology-aware facial motion.

3 Proposed Method

  • The framework models motion as horizontal spreading, vertical opening, and depth-wise protrusion, then composes the predicted directions into surface-consistent 3D facial motion.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Speech–Articulatory Memory (SAM)

  • SAM uses a contextual acoustic query to address learnable slots and retrieve short motion sequences from separate spreading, opening, and protrusion value memories.Source: §3.1 · p. 4
  • Content features cross-attend to retrieved features to decode each directional motion, while reconstruction, velocity, and window-level store losses supervise spatial accuracy and temporal coherence.Source: §3.1 · pp. 4–5

3.2 Topology-aware Articulatory Composition (TAC)

  • TAC propagates directional features across mesh neighbors with graph convolution, then applies vertex-wise temporal convolution to capture short-term dynamics while preserving spatial coordination.Source: §3.2 · p. 5
  • A learned per-vertex gate scales TAC’s residual by region sensitivity before adding it to directional motion; reconstruction, velocity, and graph-Laplacian losses train local consistency.Source: §3.2 · p. 6

4 Experiments

The experiments evaluate the method on VOCASET and TFHP using standard reconstruction and lip-articulation metrics, alongside qualitative comparisons, an ablation, trajectory analysis, and a user study. Results show strong reconstruction and articulation performance, with preferred lip sync and realism in the user study.

4.1 Datasets / 4.2 Implementation Details / 4.3 Evaluation Metrics

  • Experiments use VOCASET and TFHP, two-stage SAM/TAC training, standard reconstruction metrics, and lip-region distance and velocity measures.VOCASET has 480 sequences from 12 speakers; TFHP has 1,052 videos from 588 speakers. SAM is trained for 250 epochs, then frozen while TAC is trained for 150 epochs.Source: §4.1 · p. 6, §4.2 · p. 6, §4.3 · pp. 6–7

4.4.1 Overall Comparisons

  • The method achieves the lowest FVE, LVE, and LDTW on both datasets; upper-face dynamics are not explicitly modeled for FDD.The reported TFHP results show lip-region improvements without degrading overall reconstruction quality.Source: Table 1 · p. 7, §4.4.1 · p. 7

4.4.2 Articulatory Motion Comparisons

  • Distance errors are lowest on ICW, ILD, and LP; velocity errors are best on ILD and LP and second-best on ICW.These RMSE-based distance and velocity errors measure three lip-articulation signals.Source: Table 2 · p. 7, §4.4.2 · p. 7

4.4.2 Articulatory Motion Comparisons / 4.6.2 Directional Articulatory Modeling

  • Radar comparisons place the method closest to ground truth across directions; its trajectories also follow ground truth across spreading, opening, and protrusion, unlike direction-specific baseline deviations.The trajectory examples include attenuated protrusion at /tr/ and /v/, and noisy spreading at /aU/.Source: §4.4.2 · p. 7, Figure 3 · p. 7, §4.6.2 · p. 9

4.5.1 Sequence-Level Mesh Comparisons

  • On TFHP, phoneme-aligned mesh renderings show distinct articulatory configurations and low lip and lower-face L1 errors, while upper-face appearance remains comparable to competing methods.The figure displays renderings above per-vertex L1 error maps, color-coded from 0 to 0.015.Source: §4.5.1 · p. 8, Figure 4 · p. 8

4.5.2 Sequence-Level Visible Articulatory Dynamics

  • Slit-scans show distinct spreading and opening trajectories, with steeper slopes at transitions that match ground-truth patterns or exceed competing methods in distinctness.The horizontal and vertical scans represent lip width and mouth height over time, respectively.Source: §4.5.2 · p. 8, Figure 5 · p. 8

4.6.1 Topology-aware Composition

  • Adding TAC to SAM improves all metrics in the VOCASET ablation by composing per-direction motions through spatial coupling across the facial surface.SAM alone reaches between UniTalker and MemoryTalker on the reported Table 1 comparison.Source: §4.6.1 · p. 9, Table 3 · p. 9

4.7 User Study

  • Across seven competitors, the method receives average preferences of 70.3% for Lip Sync and 70.7% for Realism; each comparison has p < 0.0001 against chance.The study reports 24 valid participants and tests preference against a 50% chance level.Source: Table 4 · p. 9, §4.7 · p. 9

5 Conclusion

The paper frames speech-driven facial animation through structured visible articulation rather than holistic vertex regression. Its SAM and TAC components model speech-to-articulation correspondence and compose directional motions into surface-consistent facial motion under mesh topology.

5 Conclusion

  • The framework represents facial motion through structured articulatory directions, moving beyond holistic vertex regression toward physically grounded and interpretable animation.SAM models speech-articulation correspondence, while TAC composes directional motions under mesh topology into surface-consistent motion.Source: §5 · p. 9

A.1.1 Implementation Details

The appendix describes directional motion autoencoders used to produce SAM's alignment targets and the periodic positional encoding used in SAM. The autoencoders are pretrained and frozen before SAM training, while positional cycles are set to approximately one second.

Motion Autoencoder – Architecture Details

  • Three independent encoder-decoder pairs encode and reconstruct ground-truth motion, one for each articulatory direction; their encoder outputs supervise retrieved features in Eq. 5.Each pair uses a Transformer encoder and decoder, and the architecture is summarized in Table S1.Source: §A.1.1 · p. 15, Table S1 · p. 15

Motion Autoencoder – Training Details

  • For each dataset, the directional autoencoders are pretrained, then frozen so their encoder outputs supervise SAM's value memory via Lstore.Training normalizes motion over time using per-vertex z-score statistics and minimizes reconstruction and velocity losses in the raw vertex domain.Source: §A.1.1 · p. 15

SAM – Periodic Positional Encoding

  • Periodic positional encoding repeats its ρ-step embedding cycle; ρ is set to each dataset's frame rate so one cycle spans approximately one second.The reported periods are 30 frames for VOCASET and 25 for TFHP.Source: §A.1.1 · p. 15

A.1.2 Evaluation Details

The evaluation derives signed spreading, opening, and protrusion trajectories from lip anchors, then measures their directional tendencies and similarity to ground truth. It also specifies preprocessing and model settings, including a 32-slot memory selected on VOCASET.

A.1.2 Evaluation Details

  • The evaluation computes median-centered signed spreading, opening, and protrusion signals from four lip anchors, preserving motion direction.The signals use LMC/RMC x-coordinates for spreading, UL/LL y-coordinates for opening, and UL/LL z-coordinates for protrusion.Source: §A.1.2 · p. 16
  • A centered 7-frame local linear-fit slope estimates each trajectory’s trend, with reflective padding at sequence boundaries.This is a smoother estimate than a single-frame finite difference.Source: §A.1.2 · p. 16
  • Slope signs split each direction into paired tendencies: spreading/rounding, opening/closing, and protrusion/retraction.Positive slopes indicate the canonical direction; negative slopes indicate its opposite.Source: §A.1.2 · p. 16
  • Tendency strengths are pooled across evaluation sequences, compared with ground truth by a strength ratio, and scored symmetrically on a log scale.The similarity equals 1 at an exact match and decreases for either over- or underestimation.Source: §A.1.2 · p. 16
  • VOCASET meshes are downsampled to 30 fps and TFHP’s no-head-pose variant is used because the framework does not model rigid head movement.Audio is resampled to 16 kHz; TFHP meshes are decoded with zero global head rotation and translation. Table S2 summarizes metric scales and units.Source: §A.1.2 · pp. 16–17, Table S2 · p. 17
  • N = 32 achieves the lowest FVE and LVE on VOCASET and is used in all experiments.The sweep compared N ∈ {16, 32, 64, 128} with other hyperparameters fixed.Source: Figure S1 · p. 17, §A.1.2 · p. 17

A.2.1 SAM Ablations

The ablations test directional decomposition, SAM’s explicit memory and query-refinement design, and value-memory supervision. Directional decomposition and explicit memory improve reported metrics, while the query/refinement substitutions and removal of store loss degrade performance.

A.2.1 SAM Ablations

  • Directional decomposition reduces FVE by 8.4%, LVE by 11.7%, FDD by 20.5%, and LDTW by 27.1% versus the unified-branch baseline.The unified branch misses dominant protrusion for /u/, opening for /a/, and protrusion and spreading for /o/.Source: §A.2.1 · pp. 17–18, Table S3 · p. 18, Figure S2 · p. 18
  • Explicit SAM memory reduces FVE by 6.3%, LVE by 14.5%, FDD by 19.8%, and LDTW by 29.8% over the no-memory baseline.Without memory banks, the acoustic query and content feature must perform the disambiguation without stored priors.Source: Table S4 · p. 18, §A.2.1 · p. 18
  • Replacing either the acoustic query or content refinement degrades all metrics, with the largest drop when acoustic features serve both roles.The reported design uses Mel features to address phonetic-context-dependent memory slots and HuBERT features to refine retrieved motions.Source: §A.2.1 · pp. 18–19, Table S5 · p. 19
  • Removing the value-memory store loss consistently degrades all metrics and leaves memory slots without direct supervision to organize into direction-specific prototypes.The ablation retains vertex-domain reconstruction and velocity supervision.Source: §A.2.1 · p. 19, Table S6 · p. 19

A.2.2 TAC Ablations

The TAC ablations examine topology-aware composition, its internal residual connection, and graph Laplacian consistency supervision. The reported comparisons favor topology-aware composition and retaining both the residual path and Laplacian loss.

A.2.2 TAC Ablations

  • Topology-aware TAC improves over MLP composition: FVE 0.799 → 0.771, LVE 0.243 → 0.235, FDD 0.103 → 0.097, and LDTW 0.146 → 0.132.The MLP processes vertices independently, whereas GCN-based TAC couples motions across mesh-connected vertices.Source: Table S7 · p. 19, §A.2.2 · p. 19
  • Removing TAC’s internal residual connection degrades all metrics across its stacked GCN+TCN blocks.The residual adds cumulative refinement to the initial projected motion.Source: §A.2.2 · p. 19, Table S8 · p. 20
  • Adding graph Laplacian consistency loss yields consistent gains across all metrics by supervising local surface curvature.The loss targets high-frequency geometric discrepancies such as lip-corner folds and mouth creases that global vertex error may underweight.Source: §A.2.2 · pp. 19–20, Table S9 · p. 20

A.2.3 Training Strategy

The section summarizes the frozen two-stage training schedule and supplementary analyses of the temporal window, alongside reported articulatory, qualitative, generalization, user-study, and limitation findings. It also notes training on one RTX 4090 and statistically significant user-study comparisons, while stating that other training details are not described.

A.2.3 Training Strategy

  • Pretraining SAM for 250 epochs, then freezing it while training TAC for 150, outperforms unfrozen fine-tuning and joint training from scratch on VOCASET.The authors attribute the single-stage drop to SAM’s directional motion prototypes needing dedicated supervision before TAC uses them; freezing preserves direction-specific retrieval.Source: Table S10 · p. 20, §A.2.3 · p. 20

A.2.4 Hyperparameter Sensitivity

  • At Ω=7, the 15-frame temporal context yields the lowest VOCASET FVE and LVE; shorter windows lack motion context, while longer windows include less-relevant information.The study evaluates Ω from 0 to 9 and uses Ω=7 in all experiments.Source: §A.2.4 · pp. 20–21, Figure S3 · p. 21

A.3 Generalization Studies

  • On BIWI’s 23,370-vertex topology, the full model achieves the best FVE, LVE, and LDTW, generalizing beyond FLAME’s 5,023-vertex topology.The evaluation follows prior BIWI splits and uses the same optimizer, learning rate, batch size, and training schedule as VOCASET and TFHP.Source: §A.3 · p. 21, Table S11 · p. 21
  • On TFHP, our method achieves the lowest distance and velocity errors across ICW, ILD, and LP, extending VOCASET findings across speaking styles.The reported comparison covers all three signals for both distance and velocity errors.Source: Table S12 · p. 21, §A.3 · p. 21

A.4 Qualitative Analysis

  • On TFHP, TAC reduces SAM-only lower-face residual errors, while its gate allocates the highest spatial weight to lips, at 2.0× the uniform baseline.Gate allocation is unsupervised; neck, nose, and remaining face also exceed baseline, while largely static regions stay near or below it.Source: §A.4 · p. 22, Figure S4 · p. 22, Figure S5 · p. 22
  • Qualitative examples show phoneme-consistent memory retrieval and directional lip trajectories following ground truth, while comparisons show lower lip and lower-face errors than prior methods.Across VOCASET and TFHP, upper-face error remains comparable to competing methods; the illustrated trajectories include spreading, opening, and protrusion patterns.Source: §A.4 · pp. 23–24, Figure S6 · p. 23, Figure S7 · p. 23

A.5 User Study Details

  • The pairwise user study asked 24 valid participants to judge lip-sync accuracy and overall realism on randomized animation pairs across seven competitors.Each participant rated 15 randomly sampled clips per competitor and question; attention checks excluded two of 26 recruited participants.Source: §A.5 · p. 26

A.6 Limitations and Broader Impacts

  • The method lacks explicit teeth and tongue modeling and non-articulatory upper-face dynamics such as blinking; the authors also note deepfake misuse risks alongside potential beneficial applications.On TFHP, DiffPoseTalk’s FDD is 0.321 versus 0.578 for this audio-only method and 0.581 for ARTalk; on VOCASET, this method achieves the best FDD (0.097).Source: §A.6 · p. 27

NeurIPS Paper Checklist

  • All seven pairwise user-study comparisons yielded p < 0.0001 against chance, using 24 valid participant responses.Two of 26 recruits were excluded after failed attention checks; each participant rated 15 randomly sampled clips per competitor.Source: NeurIPS Paper Checklist · p. 31
  • Experiments used batch size 1 on a single NVIDIA RTX 4090 GPU with 24 GB VRAM and a 250-epoch SAM stage followed by a 150-epoch TAC stage.Source: NeurIPS Paper Checklist · p. 31
Loading 2609.30517v1…