PaperlayerView exact revision on arXiv

Source-linked AI summary

emg2face: Expressive Facial Animation with High-Density Surface EMG

Ganidhu Abey, Wendy Greening, Ashika Kamboj, Leonhard Helminger, Abhijeet Ghosh, Karel Petranek, Sergio Orts Escolano, Dinesh K. Pai

arXiv:2610.09304v1cs.GRcs.CVcs.ETcs.LG

TL;DR

Optical face capture can be obstructed by HMDs and raises privacy concerns. This paper uses synchronized HD-sEMG recordings and a staged head-model fit to predict facial expression weights from muscle activity. Predictions achieve 0.43 mm median vertex error on held-out expressions, with the hidden eye region predicted as well as the rest of the face.

  • Problem

    Optical face capture can be obstructed by HMDs, requires camera rigs, raises privacy concerns, and misses muscle activations that barely deform the face.

    Source: §1 · p. 1

  • Method

    The paper synchronizes EMG and video, fits an identity-independent expression model to facial landmarks, and trains a network to predict expression weights from two-grid HD-sEMG.

    Source: §1 · p. 1, §3.2 · p. 3, §5 · p. 10

  • Results

    0.43 mm median vertex error on held-out expressions is about half that of a face held at mean expression, and the HMD-hidden eye region is predicted as well as the rest.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    The method needs no view of the face, and its output can be applied directly to other faces and characters.

    Source: §5 · p. 11

    Predictions are available about 1.3 seconds after the movement they describe, so the non-causal processing is not suitable for real-time animation.

    Source: §5 · p. 11

Abstract

from arXiv · show

Facial movements convey subtle and important information that is critical for human social communication. Optical methods for face capture are difficult or impossible to use when the face is occluded by head-mounted devices (HMDs), such as VR headsets. Even with a clear line of sight, such methods raise privacy concerns and require head-mounted capture rigs that offset cameras and lighting from the face. We show that high-density surface electromyography (HD-sEMG) provides a viable non-optical alternative that addresses these challenges. We measured 64 EMG channels, using two textile EMG grids, with 32 from the forehead (typically occluded by an HMD) and 32 from the side of the face. EMG data were digitized at 2048 Hz and filtered. Facial movements were simultaneously recorded and used to estimate 478 3D facial landmarks using MediaPipe's Face Landmarker. A major challenge in such multimodal recordings is synchronizing EMG and video data, which have different sampling frequencies and independent clocks. We developed a novel synchronization method using analog audio bursts that is capable of sub-millisecond synchronization. We also developed a staged fitting method that fits a recent high-resolution parametric head model (GNM), with 253 identity blendshapes and 383 expression blendshapes, to the MediaPipe landmarks as participants performed different facial expressions. We trained a deep neural network comprising per-grid spatial encoders followed by a dilated temporal convolutional network (TCN) to predict blendshape parameters from HD-sEMG signals at 100 Hz. Once trained, the network can predict expression blendshapes solely from HD-sEMG recordings. The output can be rendered using standard real-time blendshape animation methods. We demonstrate the methods using recordings from 25 participants, and direct expression transfer to a variety of human faces and non-human characters.

1 Introduction

Optical face capture is cumbersome, can be obstructed by HMDs, and raises privacy concerns, motivating HD-sEMG as a non-optical way to predict facial expression. The paper combines synchronized recordings, identity-independent expression fitting, and neural prediction, demonstrating low error on held-out expressions and transfer to other characters.

  • Optical capture can require cumbersome camera-and-lighting rigs, be obstructed by HMDs, raise privacy concerns, and miss muscle activations that barely deform the face.Source: §1 · p. 1
  • The network predicts expression weights from HD-sEMG alone, using a spatial encoder for each grid followed by a dilated temporal convolutional network.The two 32-electrode textile grids are placed on the forehead and cheek.Source: §1 · pp. 1–2
  • The method is demonstrated on recordings from 25 participants and transfers expressions to other faces and non-human characters.Source: §1 · p. 2
  • Audio tones recorded by both devices synchronize EMG and video, modeling clock drift and aligning recordings to 0.15 ms RMS.Source: §1 · p. 2
  • A staged fit of the parametric head model separates head pose, eyelid closure and gaze from expression, yielding identity-independent expression weights.Source: §1 · p. 2
  • 0.43 mm median vertex error on held-out expressions is about half the error of a face held at mean expression, with the HMD-hidden eye region predicted as well as the rest.The forehead grid alone retains more than four fifths of the variance explained with both grids.Source: §1 · p. 2

2 Related Work

Prior face-capture approaches under occlusion rely on cameras or strain gauges, while facial EMG research has largely focused on discrete-state classification. Continuous facial-shape prediction remains less common, and this work uses textile HD-sEMG grids with a high-resolution parametric head model.

  • Camera-based systems for occluded faces combine strain gauges or partial camera views, which remain oblique and raise privacy concerns.Examples include Li et al., Olszewski et al., Wei et al., and commercial headset systems [BWL+24] [CWV+24].Source: §2 · p. 2
  • Facial sEMG has been recorded with multi-electrode arrays, but each electrode mixes signals from overlapping muscles, making source localization ill-posed.Compared with video action units, sEMG agrees on event timing but not consistently on source, and detects activations such as teeth clenching.Source: §2 · p. 2
  • Continuous facial-shape prediction from EMG is less common than classifying discrete states; prior examples include lip markers, expression blendshapes, and restoration of electrode-occluded faces.Eskes et al. predicted ten lip markers; Man et al. reported poor performance on a new participant; EIFER [BAGLD25] maps between FLAME parameters and muscle activity.Source: §2 · p. 2
  • This work records 64 channels with two textile grids and fits GNM, a 17,821-vertex head model with 383 expression blendshapes.Source: §2 · p. 2

3 Methods

The methods pair facial EMG and video capture with tone-based clock alignment and staged conversion of monocular landmarks into expression targets. A per-participant network maps EMG envelopes to those targets, followed by geometric correction of eyelid intersections.

3.1 Data Acquisition

  • The capture protocol records facial movement and muscle activity together using two 32-channel textile grids, a front-facing camera, and cued repeated expressions.One grid is placed on the forehead and one on the upper right cheek; participants perform movements such as eyebrow raises, blinks, and smiles.Source: §3.1 · pp. 2–3

3.2 Synchronization

  • 0.15 ms RMS (0.49 ms maximum) onset agreement was achieved after fitting clock offset and drift across matched tones.In a representative 12-minute recording, 161 tones were matched; without drift correction, the offset alone would reach 1.8 ms by the end.Source: §3.2 · p. 3

3.3 Facial Expression Encoding

  • Three fitting stages separate head pose, eyelid closure and gaze, and remaining expression, yielding 383 expression weights independent of identity and head pose.The targets are resampled to 100 Hz and can be rendered with standard blendshape animation or applied to other identities and characters.Source: §3.3 · pp. 4–5

3.4 Predicting Expressions from HD-sEMG

  • A per-participant network uses separate spatial encoders for the two EMG grids and a dilated temporal convolutional network to predict fitted expression weights.The temporal network combines 2.5 seconds of centered context; one repetition of each expression in each recording block is held out for testing.Source: §3.4 · p. 6

3.5 Correcting Eyelid Intersections

  • An eyelid correction constrains lid vertices outside a fitted eyeball shell while leaving geometry away from the eye unchanged.A fixed set of vertices is corrected throughout the sequence to avoid temporal flicker.Source: §3.5 · p. 6

4 Results

Across held-out cued expressions, HD-sEMG predicted facial motion in 25 participants, including when the face was covered or expressions were difficult to see. The results also demonstrate forehead-only decoding, expression transfer, and wrinkle synthesis.

4.1 Performance and Validation

  • 0.43 mm median vertex error—about half the 0.87 mm mean-face baseline—accompanied predictions explaining 64% of facial-motion variance, with the HMD-hidden eye region predicted as well as the rest.The network outperformed ridge regression for 23 of 25 participants, with mean r of 0.67 versus 0.55.Source: §4.1 · pp. 6–7, Figure 7 · p. 7
  • Seven lower-quality recordings had mean r of 0.65 versus 0.79 for the other 18, while brow raises, eye closure, and blinks tracked closely but speech lip and jaw motion was underestimated.The lower-quality recordings explained 54% of motion variance versus 66% for the others; the authors link the pattern to possible poor electrode contact.Source: §4.1 · p. 7

4.2 Decoding with the Face Covered

  • Under a sleep mask, predicted brows rose 1.9–2.7 mm in seven of eight brow raises, within the uncovered range, although prediction accuracy could not be measured against the hidden face.MediaPipe reported a face in 99% of masked frames but placed the brows on the mask; its tracked brow rise was 1.0 mm versus 2.3 mm without the mask.Source: §4.2 · p. 7, Figure 10 · p. 8

4.3 Decoding from the Forehead Alone

  • With the forehead grid alone, predictions explained 57% of facial-motion variance versus 64% with both grids, retaining more than four fifths of the full-grid variance while the cheek grid especially improved speech.Median vertex error rose from 0.43 to 0.50 mm; speech R2 was 0.42 with the forehead grid versus 0.62 with both.Source: §4.3 · p. 8, Figure 11 · p. 9

4.4 Decoding Affect and Invisible Expressions

  • In one participant, teeth clenching produced strong, distinct EMG despite only 0.27 mm median facial displacement; a classifier detected held-out clenches with AUC 0.999.At a threshold flagging 1% of other frames, it detected at least 96% of frames from every held-out clench; this was a single participant clenching on instruction, not under stress.Source: §4.4 · pp. 8–9

4.5 Expression Transfer

  • The same predicted expression weights were applied to nine GNM identities and to non-human characters, with registration artifacts for the squirrel while smiling.The nine heads comprise the mean GNM head and eight other identities.Source: §4.5 · p. 10, Figure 13 · p. 10, Figure 14 · p. 10

4.6 Adding Secondary Effects

  • Expression-driven wrinkles add forehead furrows and nasolabial folds, with wrinkle depth driven by skin compression across each furrow rather than deformation along it.The method uses three horizontal forehead furrows and one pair of nasolabial folds; the profile conserves displaced volume so the skin is not compressed.Source: §4.6 · p. 10, Figure 15 · p. 10

5 Conclusions

The paper demonstrates HD-sEMG-driven facial animation with a high-resolution head model, using synchronized recordings and a staged fitting and prediction pipeline. On held-out cued expressions, it reports submillimeter error while identifying latency, data, and ground-truth constraints.

5 Conclusions

  • The pipeline models clock drift to align tone onsets within 0.2 ms, separates pose, eyelid closure, gaze, and expression in a staged GNM fit, then predicts all 383 expression weights at 100 Hz.The network combines a spatial encoder for each electrode grid with a dilated temporal convolutional network.Source: §5 · p. 10
  • 0.43 mm median vertex error is about half the mean-expression face error; the eye region is predicted as well as the rest, and forehead-only input retains over four fifths of explained motion variance.The comparison baseline is each participant’s mean-expression face.Source: §5 · p. 11
  • Predictions arrive about 1.3 seconds after the movement, so the current noncausal processing is not suitable for real-time animation.The filters run forward and backward, and the network uses EMG both before and after each frame.Source: §5 · p. 11
  • Each network is trained for one participant using a synchronized session; it generalizes to cued repetitions, but not yet to free conversation or a new person without such a session.The paper identifies unscripted speech and expressions as a natural next training-data step.Source: §5 · p. 11
  • Ground truth relies on single-camera MediaPipe tracking, which captures depth poorly and provides few forehead landmarks.Source: §5 · p. 11
  • The authors describe the approach as promising for animation, XR, psychology, and medicine, and report that its output can be applied to other faces and characters.Source: §5 · p. 11

A EMG Signal Processing

The processing pipeline converts raw EMG into a synchronized, log-compressed activation envelope for network input. Filtering removes drift and mains interference, while resampling uses the video clock to avoid timing error.

A EMG Signal Processing

  • Raw EMG contains a large offset and slow drift; below 20 Hz, drift power exceeds muscle-activity power by about three orders of magnitude.The example offset and drift are several millivolts, compared with muscle activity of a few hundred microvolts.Source: A EMG Signal Processing · p. 14
  • A 20–450 Hz band-pass and 60, 120, and 180 Hz notch filters remove drift, movement artifacts, and mains interference; rectification and an 8 Hz low-pass filter form the activation envelope.Forward-and-backward filtering avoids shifting the envelope relative to video.Source: A EMG Signal Processing · p. 14
  • The envelope is linearly resampled at exact video-clock times to 100 Hz and compressed as log(1+e); keeping every 20th 2048 Hz sample would instead yield 102.4 Hz and about 160 ms drift over 7 seconds.Frames without tracked faces are excluded from training and evaluation.Source: A EMG Signal Processing · p. 14
  • During a held brow raise, the processed activity rises over the forehead grid while the cheek grid changes little.Source: A EMG Signal Processing · p. 14

B Interpreting the Vertex Error

Whole-face vertex error and R2 need caution because most frames and vertices move little, region-level motion differs, and medians can conceal failures. Regional and blink analyses show stronger lower-face motion and identify lower-signal recordings as a source of missed eyelid movements.

B Interpreting the Vertex Error

  • Submillimeter whole-face error partly reflects how little the face moves; averages also mix regions with different motion, and medians hide failures.The authors identify the error of the mean face as a more meaningful reference.Source: B Interpreting the Vertex Error · p. 14
  • Prediction roughly halves median error in both regions: 1.02 to 0.50 mm over the rest of the face and 0.68 to 0.36 mm over the eye region; the lower face contributes a median 66% of whole-face error.The lower face, especially lips and chin, moves more than the eye region.Source: B Interpreting the Vertex Error · p. 14
  • At deepest held-out closures, fitted blinks close by a median 7.5 mm and predictions by 6.7 mm; median aperture error is 0.39 mm for blinks and 0.20 mm for eye closures.Source: B Interpreting the Vertex Error · p. 14
  • Among 49 blinks, predictions closed the eye by less than half as much as the fit, or not at all; 10 such cases came from seven lower-signal recordings contributing 13 blinks.Eye closures were missed in 9 of 47 trials, 4 from those recordings. Trials with fitted closure under 3 mm are omitted from Figure 16.Source: B Interpreting the Vertex Error · p. 14, Figure 16 · p. 14
Loading 2610.09304v1…