Source-linked AI summary
Attention Mesh: High-fidelity Face Mesh Prediction in Real-time
Ivan Grishchenko, Artsiom Ablavatski, Yury Kartynnik, Karthik Raveendran, Matthias Grundmann
TL;DR
Accurate face-mesh registration is needed for applications that depend on precise eye and lip contours, while a single whole-face regressor can degrade quality in those regions. Attention Mesh uses spatial-transformer regional heads within a unified model; it is reported to run 25%+ faster than cascaded face and region models while improving eye-region quality and matching lips performance.
Problem
A single face-regression network can degrade landmark quality around perceptually important eyes and lips, while cascaded regional models add inference costs.
Source: §1 · p. 1Method
Results
25%+ faster than cascaded face and region models on a typical modern mobile device, with better eye-region quality and comparable lip performance.
Source: §3 · p. 3Takeaways & Limitations
Abstract
from arXiv · showhide
We present Attention Mesh, a lightweight architecture for 3D face mesh prediction that uses attention to semantically meaningful regions. Our neural network is designed for real-time on-device inference and runs at over 50 FPS on a Pixel 2 phone. Our solution enables applications like AR makeup, eye tracking and AR puppeteering that rely on highly accurate landmarks for eye and lips regions. Our main contribution is a unified network architecture that achieves the same accuracy on facial landmarks as a multi-stage cascaded approach, while being 30 percent faster.
1. Introduction
The paper targets accurate registration of a detailed 3D face mesh, especially around eyes and lips, without the cost of a cascade of separate regional models. Attention Mesh uses region-specific heads with spatial transformers in one model, claiming comparable quality to the cascade at up to 30 percent faster inference.
1. Introduction
- A single whole-face regression network degrades quality in perceptually important regions such as the lips and eyes.The paper directly predicts 3D face-mesh vertices rather than using a parametric face model.Source: §1 · p. 1
- Region-specific heads with spatial transformers let one model match cascaded quality while running up to 30 percent faster at inference.The authors also say the unified architecture is easier to train and distribute than chained, disparate networks.Source: §1 · p. 1
- Cascaded models improve regional accuracy by cropping the initial mesh prediction and passing those crops to specialized networks, but add large models and costly GPU–CPU synchronization.The paper builds on a detector-plus-landmark-regression architecture [5].Source: §1 · p. 1
- The authors describe their implementation as related to work combining spatial transformers and salient-region heads, while noting that work pursued a different goal.They focus on landmarks for eyes, irises, and lips, and report quality and inference benchmarks.Source: §1 · p. 1
2. Attention mesh
Attention Mesh branches from a shared image feature map into a whole-face mesh predictor and regional heads for salient contours. Spatial transformers extract region features, and two-phase training adapts those heads from ideal crops to model-predicted crops.
2. Attention mesh
- The network uses a 64×64 feature map: one submodel predicts 478 3D landmarks and region crop bounds, while other submodels use attention-extracted 24×24 regional maps.The model accepts a 256×256 image supplied by a face detector or tracking from a previous frame.Source: §2 · pp. 1–2
- The model targets the lips and both eyes; each eye submodel separately predicts the iris at 6×6 resolution, reusing eye features while keeping the iris output independent.Separate submodels let the architecture allocate capacity by region, and eye and lip features are normalized for horizontal alignment and uniform size.Source: §2 · p. 2
- Training first uses augmented ground-truth crops to train submodels independently, then uses model-predicted crop locations to adapt the regional submodels.Source: §2 · p. 2
- A spatial transformer extracts 24×24 region features from the 64×64 map using an affine transformation that can zoom, rotate, translate, or skew the sampled grid.The transformation can be supervised or computed from the face-mesh submodel output.Source: §2 · p. 2
- The dataset contains 30K in-the-wild mobile-camera photos; manual 2D annotations emphasized consistent salient contours, and a synthetic model approximated the Z coordinate.Source: §2 · p. 3
3. Results
On a typical modern mobile device, Attention Mesh is reported to run faster than cascaded face and region models while matching or improving their regional landmark quality. The evaluation compares inference performance and mean normalized error.
3. Results
- The cascade baseline consists of independently trained base-mesh, eye, and lip models run in succession.Source: §3 · p. 3
3. Results — Performance
- 25%+ faster than the cascade of separate face and region models on a typical modern mobile device.The comparison uses TFLite GPU inference; a further 5% speed-up is attributed to fewer CPU–GPU synchronizations because inference runs in one GPU pass.Source: §3 · p. 3, Table 1 · p. 3
3. Results — Mesh quality
- Attention Mesh outperforms the cascade on eye regions and demonstrates comparable performance on lips.Table 2 reports mean normalized error in 2D; the paper defines its representative metric using distance from predicted to ground-truth points, normalized by interocular distance or region-corner distance.Source: §3 · p. 3, Table 2 · p. 3
4. Applications
The paper presents real-time face-mesh prediction as supporting AR makeup and virtual puppeteering. Its examples use lip contours for lipstick rendering and mesh-derived outputs for facial triggers.
4. Applications
- The model's real-time performance enables applications including virtual makeup try-on and puppeteering.Source: §4 · p. 3
4. Applications — AR Makeup
- The lipstick renderer uses lip-submodel contours; in an A/B test with 10 images and 80 people, 46% of AR samples were classified as real and 38% of real samples as AR.The paper notes that small alignment errors can make AR makeup appear uncanny.Source: §4 · p. 3
4. Applications — Puppeteering
- For puppeteering, attention-mesh outputs feed a small fully connected model predicting 10 mouth and 8 eye blend-shape coefficients per eye.Laplacian mesh editing morphs a canonical mesh to the predicted mesh so the coefficients can be used across faces without additional fine-tuning.Source: §4 · p. 3
5. Conclusion
The unified model uses differentiable attention to allocate computation to salient facial regions, enabling accurate face mesh prediction in real time without the performance penalty of independent region-specific models.
- Differentiable attention directs computational resources to salient facial regions without the performance penalty of independent region-specific models.Source: §5 · p. 3
- The model and demos were planned for availability in MediaPipe.Source: §5 · pp. 3–4