PaperlayerView exact revision on arXiv

Source-linked AI summary

IMU2Face: Real-time Gesture-driven Facial Reenactment

Justus Thies, Michael Zollhöfer, Matthias Nießner

arXiv:1801.01446v1cs.CV

TL;DR

IMU2Face addresses gesture-driven control of facial expressions in a target video by combining facial reenactment with inertial sensing. It analyzes the target with Face2Face and uses a hand-mounted IMU to modify expressions, demonstrating real-time adaptation based on intuitive gestures.

  • Problem

    IMU2Face targets facial-expression control in a target video using intuitive hand gestures rather than source-video expression transfer.

    Source: §2 · p. 1, §4 · p. 1

  • Method

    The system analyzes the target actor with Face2Face and uses a hand-mounted IMU’s orientation to control the target actor’s facial expressions.

    Source: §2 · p. 1, §3 · p. 1

  • Results

    IMU2Face demonstrates real-time adaptation of a target actor’s expressions based on intuitive hand gestures.

    Source: §4 · p. 1

  • Takeaways & Limitations

    IMUs provide an easily applicable basis for real-time gesture tracking and may drive other actions.

    Source: §4 · p. 1

Abstract

from arXiv · show

We present IMU2Face, a gesture-driven facial reenactment system. To this end, we combine recent advances in facial motion capture and inertial measurement units (IMUs) to control the facial expressions of a person in a target video based on intuitive hand gestures. IMUs are omnipresent, since modern smart-phones, smart-watches and drones integrate such sensors, e.g., for changing the orientation of the screen content, counting steps, or for flight stabilization. Face tracking and reenactment is based on the state-of-the-art real-time Face2Face facial reenactment system. Instead of transferring facial expressions from a source to a target actor, we employ an IMU to track the hand gestures of a source actor and use its orientation to modify the target actor's expressions.

1 Related Work

IMU2Face builds on prior systems for inertial body tracking and real-time facial reenactment, including Face2Face for in-the-wild color videos.

  • Sparse Inertial Poser reconstructs full-body motion using six body-mounted IMUs [5].Source: §1 · p. 1
  • Face2Face enabled real-time facial reenactment in standard in-the-wild color videos.Source: §1 · p. 1
  • Earlier systems used RGB-D sensing for live expression transfer, while FaceVR addressed HMD removal through self-reenactment in VR [4].Source: §1 · p. 1

2 Method Overview

The system analyzes a target video with Face2Face, then uses a hand-mounted IMU to control the target actor’s jaw motion instead of transferring expressions from a source video.

  • Given a target actor’s video, Face2Face analyzes facial geometry and expressions to obtain a 3D actor model, enabling editing of facial expressions in the target video.Source: §2 · p. 1
  • An IMU attached to the source actor’s hand controls the target actor’s jaw motion.Source: §2 · p. 1

3 Motion Transfer

Motion transfer converts hand-held IMU orientation into target facial expressions by optimizing jaw-region deformation in real time.

  • The system computes expression blendshape weights from IMU orientation by minimizing deformation-transfer energy over the jaw region.Source: §3 · p. 1
  • A reference IMU pose initializes the mapping, after which the resulting linear system is solved for every frame.Source: §3 · p. 1
  • An RC drone flight controller filters and streams IMU data at high rates to a desktop PC using BetaFlight.Source: §3 · p. 1

4 Conclusion

IMU2Face demonstrates gesture-driven adaptation of a target actor’s facial expressions and highlights IMUs’ applicability to real-time tracking.

  • IMU2Face adapts a target actor’s expressions based on intuitive hand gestures.Source: §4 · p. 1
  • The authors identify easy applicability of IMUs for real-time tracking and suggest similar strategies could drive other actions.Source: §4 · p. 1
Loading 1801.01446v1…