Source-linked AI summary
Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs
Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, Matthias Grundmann
TL;DR
The paper addresses estimating 3D facial-mesh vertex positions from single-camera RGB input for mobile AR. It develops a neural-network model with a 468-point mesh and reports deployment in major mobile AR applications and developer APIs.
Problem
Estimating 3D facial-mesh vertex positions from single-camera input matters for face-based AR effects without requiring depth-sensor information.
Method
The method uses a neural network that estimates 468 independently treated mesh-vertex positions from single RGB-camera frames or streams, with full and lightest variants targeting mobile GPUs and CPUs.
Results
The technology described drives major AR self-expression applications and AR developer APIs on mobile phones.
Takeaways & Limitations
The 468-point mesh supports plausible smooth facial surfaces for expressive effects, virtual try-on, makeup, and related AR rendering.
Takeaways & Limitations
Predicted depth is not metrically accurate because z-coordinates receive supervision only from synthetic 3D rendering, although the resulting meshes are visually plausible.
Abstract
from arXiv · showhide
We present an end-to-end neural network-based model for inferring an approximate 3D mesh representation of a human face from single camera input for AR applications. The relatively dense mesh model of 468 vertices is well-suited for face-based AR effects. The proposed model demonstrates super-realtime inference speed on mobile GPUs (100-1000+ FPS, depending on the device and model variant) and a high prediction quality that is comparable to the variance in manual annotations of the same image.
1. Introduction
The paper frames facial geometry estimation as predicting a relatively dense 3D mesh for AR from a single RGB camera, rather than only a small set of landmarks. The model targets real-time mobile inference and supports plausible smooth face surfaces.
- The proposed task estimates positions of 468 3D mesh vertices with a neural network, treating each vertex as an independent landmark.
- The fixed-quads topology allocates higher point density to facial regions expected to vary more and matter more perceptually.The resulting mesh can form a plausible smooth surface using Catmull-Clark subdivision.
- The model takes a frame or stream from a single RGB camera without requiring depth sensor information.
- The system targets real-time mobile GPU inference and includes lighter variants for mobile CPUs without suitable GPU support.The GPU-targeting model is called the “full” model, while the CPU-oriented variant is the “lightest” model.
2. Image processing pipeline
The image-processing pipeline detects and aligns a face before mesh prediction, then maps predicted 3D coordinates back to the original image. During tracking, previous-frame crops reduce repeated detector use.
- A lightweight face detector produces a bounding rectangle and landmarks used to rotate the facial crop so the eye-center line is horizontal.
- The aligned crop is resized to 256×256 pixels for the full model or 128×128 pixels for the smallest model before mesh prediction.
- The mesh network outputs 3D landmark coordinates that are mapped back to the original image coordinate system.A separate face-flag output estimates whether a reasonably aligned face is present in the crop.
- In face-tracking mode, the detector is used only on the first frame and during rare re-acquisition events after the face-flag probability falls below a threshold.
- Centering and aligning faces lets the mesh network reserve representational capacity for prediction instead of substantial rotation and translation handling.The setup also permits reducing related augmentations while gaining prediction quality.
3. Dataset, annotation, and training
Because dense 3D mesh annotation is labor-intensive and ambiguous, the paper bootstraps supervision from synthetic renderings and sparse real-image contours, then iteratively refines suitable predictions.
- The training data comprise around 30K in-the-wild mobile-camera photos from varied sensors and changing lighting conditions.
- Training augments the photos with standard image transformations, modeled sensor noise, and randomized nonlinear intensity-histogram transformations.The histogram transformation is used to simulate marginal lighting conditions.
- Ground truth for 468 3D mesh points is labor-intensive and highly ambiguous to obtain manually.
- The initial model combines synthetic 3D morphable-model renderings with annotated 2D landmarks on semantic contours from real in-the-wild images.Synthetic renderings provide predefined correspondence for the 468 mesh points, while real annotations supervise a smaller contour-related subset.
- Up to 30% of images produced predictions suitable for refinement after the first model was trained.Subsequent iterations filtered images with tolerable prediction error and bootstrapped updated x- and y-coordinate annotations.
- The refinement brush supports large strokes followed by local adjustments while preserving mesh surface smoothness.The z-coordinates remain supervised only by synthetic 3D rendering.
4. Model architecture
The mesh predictor uses a residual architecture that subsamples aggressively early and concentrates computation in shallow layers. Its receptive fields expand quickly, supporting high-level mesh completion before final coordinate output.
- The custom residual network aggressively subsamples in early layers and dedicates most computation to its shallow part.
- Early large receptive fields allow deeper neurons to distinguish spatially relevant features such as mouth-related and eye-related patterns.Convolution padding also makes relative image location implicitly available to the network.
- The model can complete slightly occluded faces or faces crossing the image boundary.The authors associate this behavior with building a high-level, low-dimensional mesh representation that becomes coordinates only in the final layers.
5. Filtering for temporal consistency in video
Single-frame predictions can exhibit human-noticeable temporal jitter, so the pipeline applies a one-dimensional filter independently to each landmark coordinate. The filter uses recent samples and face-size-adjusted velocity estimates to produce sequences without visible jitter.
- Temporal inconsistency: Single-frame processing passes only the rotated facial bounding rectangle between frames, allowing landmark trajectories to fluctuate visibly.The fluctuations arise from changes in view, head pose, lighting, and camera sensor noise.
- Filtering approach: The proposed solution applies a one-dimensional temporal filter independently to each predicted landmark coordinate.It is inspired by the 1 Euro filter’s trade-off between noise reduction and phase-lag elimination.
- Filtering approach: A few timestamped samples provide velocity estimates that are adjusted by face size to accommodate scale changes in video streams.
- Result: The filter produces human-appealing video prediction sequences without visible jitter.
6. Results
The evaluation measures normalized 2D vertex error against manual annotation variability on a geographically diverse image set, alongside model performance characteristics. The technology also supports facial texture painting and AR object rendering on mobile phones.
- Evaluation: The evaluation uses mean absolute distance normalized by interocular distance, reporting 2D-only errors because depth comes exclusively from synthetic supervision.Interocular distance is defined using eye centers estimated as midpoints of eye-corner connecting segments.
- Evaluation: 2.56% was the estimated IOD-normalized mean absolute distance between annotations from 11 trained annotators across 58 images.
- Evaluation: The evaluation set contains 1.7K geographically diverse images, with speed estimates based on the TensorFlow Lite GPU framework.
- Performance: Table 1 reports model performance characteristics.
- Applications: The technology drives major AR self-expression applications and developer APIs on mobile phones.Examples include facial texture painting and AR object rendering with glasses.