PaperlayerView exact revision on arXiv

Source-linked AI summary

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

Danzel Serrano, Przemyslaw Musialski

arXiv:2610.03436v1cs.GRcs.CVcs.MM

TL;DR

Speech-driven facial animation can reproduce recognizable mouth poses yet simplify the motion between them, which carries coarticulation. This paper measures lip-path shaping relative to a consonant-aware route and finds flatter trajectories across four methods, with viewer judgments supporting the measured direction.

  • Problem

    Recognizable, speech-timed mouth poses can still conceal unnatural motion, while common geometry and lip-sync measures do not isolate the shape of paths between speech targets.

    Source: §1 · p. 1

  • Method

    The paper measures lip-path length against the shortest route through vowel, consonant, and vowel positions, using forced alignment and a ratio invariant to uniform motion gain.

    Source: §1 · p. 1

  • Results

    All four methods trace flatter lip paths than captured speech; DiffPoseTalk, ARTalk, and FaceFormer show clear primary deficits, while CodeTalker is marginal on the primary measure and clear on a companion statistic.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Trajectory shaping is a concrete target for improving synthesized articulation, with reference-matching training or calibrated motion correction proposed for future testing.

    Source: §7 · pp. 7–8

    The ratio ignores duration and motion timing, so paths with different anticipatory or carryover timing can receive the same score; tongue motion is also unobserved.

    Source: §7 · p. 7

Abstract

from arXiv · show

Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.

1 Introduction

Recognizable, speech-timed mouth poses can still conceal unnatural motion because the paths between them carry coarticulation. The paper introduces a consonant-aware path measure and uses it to identify flatter trajectories in synthesized speech, with controlled damping and viewer judgments supporting the measured direction.

1 Introduction

  • Recognizable mouth poses can still move unnaturally because the trajectories between poses carry articulation that individual poses do not convey.A /m/ between spread vowels closes the lips along a path shaped by its neighboring vowels.Source: §1 · p. 1
  • The measure compares lip-path length with the shortest route through vowel, consonant, and vowel positions, isolating path shape rather than motion size.It is invariant to uniform motion gain and requires only forced alignment.Source: §1 · p. 1
  • A measurable trajectory-shaping deficit provides a target for training and motion control: preserve the trajectory shaping found in real speech.The paper describes its contributions as three-fold.Source: §1 · p. 2
  • All four evaluated methods trace flatter lip paths than captured speech; DiffPoseTalk, ARTalk, and FaceFormer show clear primary deficits, while CodeTalker is marginal on that measure.CodeTalker shows a clear deficit on a companion statistic.Source: §1 · p. 2
  • In a pre-registered study, 97 viewers made 3,523 judgments, and viewers penalized damping along the same measured direction.Viewers preferred real speech in 73.4% of sentence comparisons and almost two to one on single words.Source: §1 · p. 2

2 Related Work

Prior work addresses coarticulation in facial animation, speech-animation evaluation, and motion perception, but the cited approaches do not measure residual lip-path shaping. Speech-science accounts also motivate testing how consonants and boundaries shape those paths.

2 Related Work

  • Facial-animation systems modeled coarticulation with overlapping dominance functions, recorded triphones [Bregler et al. 1997], learned target spread, and dynamic visemes.Later work exposes independent jaw and lip controls and editable factors in learned systems [Peng et al. 2023b].Source: §2 · p. 2
  • The four evaluated representatives use continuous vertex offsets in FaceFormer [Fan et al. 2022], a discrete codebook in CodeTalker [Xing et al. 2023], diffusion in DiffPoseTalk [Sun et al. 2024], and residual motion tokens in ARTalk [Chu et al. 2025].They represent different motion approaches, including real-time and offline methods.Source: §2 · p. 2
  • Lip-sync, lip-vertex error, and motion-amplitude measures address properties other than residual lip-path shape; the proposed ratio needs forced-alignment timings rather than matched ground truth.Its validation is comparative, using captured speech on the same tokens.Source: §2 · p. 2
  • The figure contrasts endpoint-only and consonant-waypoint routes: the route-based ratio distinguishes the bent /m/ path (1.98) from /d/ (1.06), whereas endpoint R reverses them (2.22 versus 2.67).The caption describes real lip trajectories with illustrative V1, C, and V2 mouth poses.Source: Figure 3 · p. 3
  • Prior work examines how rendering affects motion perception and measures effects of damped and exaggerated facial motion.This paper tests controlled motion changes, preferences for synthesized speech, and whether preferences scale with the measured deficit.Source: §2 · p. 3
  • The degree-of-articulatory-constraint account predicts that sounds resist contextual reshaping on articulators used for their constriction, while coarticulation attenuates across word and prosodic boundaries [Cho 2004, 2008; Hardcastle and Hewlett 1999].The paper tests the /hh/ prediction and examines word scope using full 3D lip-region trajectories.Source: §2 · p. 3

3 Background: Coarticulation

Coarticulation is the context-dependent shaping of continuous mouth movement: neighboring sounds overlap rather than being produced as isolated poses. The paper focuses on visible lip-region motion and treats the path between speech targets as part of speech production.

3 Background: Coarticulation

  • Coarticulation arises as movements for successive sounds overlap, so each sound’s articulation depends on its context.Anticipation is preparation for an upcoming sound while completing the current one; carryover is a completed sound’s trace on following motion.Source: §3 · p. 3
  • In “too,” the lips narrow toward “oo” during “t” and remain narrowed during the following “m”; in “please,” they spread toward “ee” during “p” and “l.”The mouth is narrower during “t” and wider during “l” than before each consonant begins.Source: §3 · p. 3
  • The paper simplifies speech-science accounts to visible lip-region motion because a talking face exposes only part of the articulatory system.The broader account tracks the tongue, jaw, and larynx directly.Source: §3 · p. 3
  • In the cited account, a consonant perturbs an underlying vowel-to-vowel gesture that continues throughout the sequence.The resulting path carries production structure; its shape, rather than travel distance, distinguishes real from concatenated motion.Source: §3 · p. 3

4 Measuring Lip-Trajectory Shaping

The section defines a consonant-aware ratio for measuring lip-path shaping and tests its behavior across consonants. It contrasts this route with the endpoint chord and identifies sampling and interpretation boundaries.

4 Measuring Lip-Trajectory Shaping

  • VCV tokens span vowel–consonant–vowel sequences, are resampled to normalized time, and use lip-vertex positions relative to a neutral template.Phone boundaries come from the Montreal Forced Aligner; VV tokens provide the naive measure’s baseline.Source: §4 · p. 3
  • /hh/ ranks twentieth of 21 under the route through C, versus second under the endpoint chord, matching its weak oral target.The endpoint-chord result is vulnerable to degeneracy when flanking vowels nearly coincide; /hh/ also ranks last when endpoints are taken at token boundaries.Source: Figure 4 · p. 4, §4 · p. 4
  • R_pw divides lip-path length by the shortest route through vowel-center, consonant-center, and vowel-center positions on the trajectory.This denominator accounts for obligatory transit through the consonant waypoint rather than counting it as detour.Source: §4 · p. 4
  • The endpoint chord counts obligatory consonant transit as detour and can inflate R when flanking vowels nearly coincide.It also confounds interior excursion magnitude with residual path shape.Source: §4 · p. 4
  • Uniformly scaling a trajectory leaves R_pw unchanged, but its value depends on sampling density, so method comparisons require frame-rate-matched ground truth.Resampling ground truth from 60 fps lowers R_pw by 0.076 at 30 fps and 0.101 at 25 fps.Source: §4 · p. 4
  • R_pw measures residual lip-path shaping beyond the three knots, not absolute coarticulation or gesture magnitude.The waypoint excursion includes both the consonant gesture and its displacement by neighboring vowels.Source: §4 · p. 4
  • Labials do not exceed non-labials under R_pw, and individual-phone rankings are unsupported because intervals overlap and ordering depends on the measure.The class difference is −0.041 [−0.071, −0.014]; the two measures’ orderings have Spearman ρ = 0.46.Source: §4 · pp. 4–5

5 The Deficit and Its Calibrated Magnitude

Against frame-rate-matched ground truth, three methods have clear deficits in residual lip-path shaping, while CodeTalker is marginal on the primary measure and clear on a companion statistic. A controlled articulation ladder calibrates the deficits; sensitivity analyses support a deficit relative to real speech, though method ordering is unstable across constructions.

5.2 Scope and Sensitivity

  • Every endpoint-based ΔR gap interval includes zero, with only 71 ground-truth and 40 method VV tokens; the small vowel-to-vowel baseline leaves that comparison underpowered.The fixed off-path medoid also loses gain invariance and reverses the amplitude-normalized comparison, showing the consequence of allowing amplitude differences into the baseline.Source: §5 · p. 5, §5.2 · p. 6

5 The Deficit and Its Calibrated Magnitude

  • Three methods have clear R_pw deficits against matched ground truth; CodeTalker’s primary gap is +0.010 [−0.001, +0.040], while its companion-statistic gap is +0.031 [+0.019, +0.057].The comparison uses 183 matched tokens from 38 utterances; Table 1 reports paired gaps and beta-equivalents.Source: Table 1 · p. 5, §5 · p. 5

5.1 Calibrating the Deficit with Controlled Articulation

  • The controlled ladder scales the fast residual by beta, from damping below 1 to exaggeration above 1, while beta=1 reproduces captured motion.Temporal Gaussian smoothing separates slow carrier and fast residual; Table 2 measures the ladder after decimation to 30 fps.Source: §5.1 · p. 5, Table 2 · p. 5
  • Both gain-invariant statistics rise monotonically with beta, but the manipulation changes shape and size: amplitude grows 2.0-fold and path length 2.5-fold as R_pw rises from 1.05 to 1.24.Preference along this ladder therefore tests the combined consequence of changing the fast component, not shape alone.Source: §5.1 · p. 5, Table 2 · p. 5
  • DiffPoseTalk (beta≈0.41) and ARTalk (0.56) correspond to real speech with roughly half its fast component removed; FaceFormer is beta≈0.85 and CodeTalker (0.92) is not separable from full articulation.These are R_pw equivalences, not claims that the models literally low-pass their motion.Source: §5.1 · p. 6

5.2 Scope and Sensitivity

  • Within-word R_pw gaps clear zero for all four methods; across boundaries, three clear zero, while CodeTalker’s interval includes zero in the smaller n=56 stratum.Ground-truth R_pw is similar within and across words, so boundary attenuation may chiefly affect articulation magnitude rather than residual path shape.Source: §5.2 · p. 6
  • Across 32 method–construction–statistic cells, ground truth exceeds the method with a clear interval in 31; the exception is CodeTalker under the primary construction.Method ordering reproduces in only a minority of constructions, while the deficit relative to real speech is the stable finding.Source: §5.2 · p. 6

6 Perceptual Evidence

A pre-registered study tested whether viewers notice motion differences using paired lower-face animations under identical audio. Viewers penalized damped real articulation, showed no detected penalty for exaggeration over the tested range, and preferred real motion in sentence comparisons; the study did not establish a word-level preference predictor.

6 Perceptual Evidence

  • The pre-registered study fixed its hypotheses, models, exclusions, and sample sizes before collection and used paired animations with the same generic face and audio.Participants rated which lower-face animation better matched the audio on a seven-point scale; 97 usable sessions contributed 3,523 judgments.Source: §6 · pp. 6–7
  • Sentence and word comparisons differ in scope: sentence stimuli included CodeTalker, DiffPoseTalk, and ARTalk, while word stimuli included all four methods.The sentence stimuli predated FaceFormer’s addition to the evaluation.Source: §6 · p. 6
  • Against damped motion, real articulation scores 5.34 (95% CI [5.06, 5.61]) at beta=0 and 4.31 ([4.03, 4.58]) at beta=0.5; exaggeration at beta=1.5 and beta=2 shows no detectable departure from neutrality.The asymmetric response does not support a preference peak at the real setting; H2, which required all four rungs to exceed neutrality, was not supported.Source: §6 · p. 7, Figure 7 · p. 7
  • Damping lowers R_pw in the direction of the evaluated methods’ shape deficits, linking the geometric finding to a controlled perceptual response without isolating shape as the sole cause.The manipulation changes both trajectory shape and amplitude.Source: §6 · p. 7
  • Real motion is preferred in 237 of 323 sentence trials (73.4%; 95% CI [68.2, 78.1]), while the study does not establish a word-level preference predictor.The pooled word-level coefficient was −0.033 (SE 0.034, 95% CI [−0.099, +0.033]); H1 was not supported.Source: §6 · p. 7
  • The score did not predict individual word-level judgments, so these results do not support using it as a proxy for viewer judgments during method development.The pooled relationship was not established even though aggregate preferences separated real and synthesized motion.Source: §6 · p. 7

7 Implications and Limitations

The evaluated animations lose residual lip-path shaping, and controlled suppression of real articulation moves the measure in the same direction as the loss. The section also bounds what the measure and study establish, and outlines ways to test and use the findings.

Implications

  • The controlled study links flatter trajectories to a perceptual cost: suppressing real articulation moves the measure in the same direction and is penalized.Together, the calibration and study make trajectory shaping a concrete target for improving speech-driven 3D facial animation.Source: §7 · p. 7

Limitations

  • The ratio is comparative, not absolute; it ignores motion duration and timing, leaving anticipatory and carryover patterns indistinguished.It observes lip-region motion only, and the consonant analysis supports the predicted /hh/ reversal rather than a complete phone ranking.Source: §7 · p. 7

Implications

  • Longer sequences could add one waypoint per interior phone, but comparisons would need to hold sequence length fixed.A timing-sensitive companion could measure when each waypoint is reached to distinguish anticipatory from carryover effects.Source: §7 · p. 7

Limitations

  • The study holds a generic, non-photorealistic face fixed within each trial, so it isolates motion rather than testing appearance variation.Whether deficit size or visibility varies across face morphologies, genders, or speaker groups remains unanswered.Source: §7 · pp. 7–8

Implications

  • Sentence comparisons favor real motion 65–92% per method, while word-level margins are narrower and vary from parity for DiffPoseTalk to 68% for ARTalk.The figure reports 237 real-motion wins among 323 sentence trials and 17.0% ties on words; FaceFormer was not in the sentence arm.Source: Figure 8 · p. 8

Limitations

  • The evaluation uses VOCASET read English; its values require validation across corpora, languages, capture setups, and speaking styles.VOCASET is the operational reference, and capture, registration, and fitting may smooth motion, potentially understating deficits.Source: §7 · p. 8
  • Matching the score exactly moves lips by roughly 8% of the method-to-ground-truth distance, limiting how much of the discrepancy it captures.Word-level prediction and the lack of a detected penalty for exaggeration also constrain direct perceptual interpretation of an individual score.Source: §7 · p. 8

Implications

  • A training objective could match generated and reference trajectory-shaping scores, while calibrated postprocessing could adjust articulation toward the reference range.Neither intervention was evaluated; the proposed test compares modified and baseline systems on held-out speech using geometric deficit and sentence-level preference.Source: §7 · p. 8

8 Conclusion

The measure finds flatter lip trajectories in all four evaluated methods, with clear primary deficits for three and a marginal primary deficit for CodeTalker. Controlled damping of real articulation is penalized, and the authors identify reference-matching training and calibrated correction as next steps to test.

Conclusion

  • All four methods flatten real lip trajectories; DiffPoseTalk, ARTalk, and FaceFormer show clear primary deficits, while CodeTalker is marginal on the primary statistic and clear on the companion.Viewers penalize damping along the same measured direction, connecting the geometric finding to perceptually relevant motion.Source: §8 · p. 8
  • Reference-matching training and calibrated motion correction are proposed next steps, to be tested on held-out speech using geometric change and sentence-level preference.The conclusion presents these as future work, not interventions evaluated in this paper.Source: §8 · p. 8
  • The articulation dial compares captured speech with its fast component removed (β=0), unmodified (β=1), or doubled (β=2).The examples use the same source frame, identity, and view, and are assessed in the controlled-articulation study arm.Source: Figure 9 · p. 9

A Perceptual Study Results

The table caption specifies the perceptual-study sample, response reporting, and analysis conventions.

Perceptual Study Results

  • After pre-registered exclusions, the study includes 97 usable sessions and 3,523 judgments.Pooled rows summarize preceding rows and add no trials.Source: Table 3 · p. 11
  • Response cells report counts and percentages of all trials, including ties; “Other” denotes altered clips in the controlled-articulation arm and method outputs in the sentence and word arms.The caption does not provide the table's individual cell values.Source: Table 3 · p. 11
  • The caption defines g as a seven-point preference score with 4 neutral, notes standardized predictors for the word-level coefficient, and says the sentence interval is binomial.FaceFormer was not included in the sentence arm.Source: Table 3 · p. 11
Loading 2610.03436v1…