Source-linked AI summary
MediaPipe Hands: On-device Real-time Hand Tracking
Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, Matthias Grundmann
TL;DR
The paper addresses real-time hand tracking for AR/VR on commodity devices without specialized hardware. It combines palm detection with landmark regression in a MediaPipe pipeline, achieving real-time performance across platforms and releasing the system openly.
Problem
The problem is real-time hand tracking for AR/VR without depth sensors or other specialized hardware on commodity mobile devices.
Method
The method combines a palm detector and hand landmark model, using temporal tracking and complementary real-world and synthetic training data.
Results
The pipeline achieves real-time performance on multiple platforms and predicts 2.5D landmarks without specialized hardware.
Takeaways & Limitations
The open-source pipeline can be deployed on commodity devices and used to build gesture-control and creative AR/VR applications.
Abstract
from arXiv · showhide
We present a real-time on-device hand tracking pipeline that predicts hand skeleton from single RGB camera for AR/VR applications. The pipeline consists of two models: 1) a palm detector, 2) a hand landmark model. It's implemented via MediaPipe, a framework for building cross-platform ML solutions. The proposed model and pipeline architecture demonstrates real-time inference speed on mobile GPUs and high prediction quality. MediaPipe Hands is open sourced at https://mediapipe.dev.
1. Introduction
MediaPipe Hands addresses mobile AR/VR hand-tracking constraints with an efficient, hardware-free pipeline that runs in real time and predicts 2.5D pose from RGB input. It is also released as an open-source, cross-platform solution.
- Hand tracking supports natural interaction and communication in AR/VR applications.
- The pipeline tracks multiple hands in real time on mobile devices using an efficient two-stage design.
- It predicts 2.5D hand pose using only RGB input, without additional specialized hardware.
- The hand-tracking pipeline is open source and supports Android, iOS, Web, and desktop PCs.
2. Architecture
The architecture combines a palm detector with a landmark model, using cropped regions and temporal tracking to support efficient mobile inference. The landmark model predicts 21 2.5D landmarks alongside hand presence and handedness outputs.
- 2. Architecture: The two-stage pipeline detects palms in the full image, then predicts high-fidelity 2.5D landmarks inside the cropped hand region.The detector supplies an oriented hand bounding box to the landmark model.
- 2. Architecture: Reusing the previous frame’s landmark-derived bounding box avoids running the palm detector on every frame.The detector runs on the first frame or when tracking indicates that the hand has been lost.
- 2.1. BlazePalm Detector: The palm detector targets scale variation, occlusion, and difficult hand appearance using palm-specific boxes, scene-context features, and focal loss.Palm boxes simplify detection because palms are more rigid, can use square boxes, and reduce the number of anchors by 3∼5.
- 2.2. Hand Landmark Model: The landmark model regresses 21 coordinates and also outputs hand presence and binary handedness classifications.The 21 landmarks contain x, y, and relative depth; the hand-presence score can trigger detector-based tracking reset.
- 2.2. Hand Landmark Model: The landmark model learns 2D pose from real and synthetic images, while relative depth is learned only from synthetic images.The model uses a shared feature extractor with three output heads and supports lighter or heavier variants for different hardware and accuracy needs.
3. Dataset and Annotation
The training data combines complementary real-world and synthetic datasets to cover appearance, gesture articulation, and depth supervision. The palm detector uses only in-the-wild data, while the landmark model uses all datasets.
- 3. Dataset and Annotation: The in-the-wild dataset contains 6K diverse images, while the in-house gesture dataset contains 10K images from 30 people.The datasets complement one another through appearance diversity and coverage of physically possible hand gestures, despite their respective limitations.
- 3. Dataset and Annotation: The synthetic dataset expands possible hand-pose coverage and provides additional depth supervision.Synthetic images use projected ground-truth 3D joints for landmark supervision.
- 3. Dataset and Annotation: The palm detector is trained only on the in-the-wild dataset, whereas all datasets train the hand landmark model.Real images receive 21-landmark annotations, while synthetic images use projected ground-truth 3D joints.
4. Results
Experiments evaluate dataset composition and model capacity for the hand landmark model, finding that real-world and synthetic data together perform best and that the Full model balances quality and speed.
- Dataset evaluation: Real-world and synthetic datasets together provide the best hand landmark model results.Evaluation is conducted on real-world images, and the synthetic data also reduces visible frame-to-frame jitter.
- Model capacity: The Full model provides a good trade-off between quality and speed.Increasing capacity further yields only minor quality improvements while significantly reducing speed.
- Dataset evaluation: Table 2 reports results for models trained from different datasets.The comparison concerns dataset composition for the hand landmark model.
- Model capacity: Table 3 reports hand landmark model performance characteristics.These characteristics support comparison of model quality and speed.
5. Implementation in MediaPipe
MediaPipe implements hand tracking as a modular graph whose detector runs only when needed, while previous-frame landmarks track the hand between detections.
- MediaPipe graph: MediaPipe represents hand tracking as a directed graph of modular Calculator components.Calculators handle model inference, media processing, and data transformations, with GPU acceleration used for operations including cropping and neural-network computation.
- Tracking optimization: The palm detector runs only when needed rather than on every video frame.The current hand location is derived from previous-frame landmarks, and detection is reapplied when hand-presence confidence falls below a threshold.
6. Application examples
The predicted hand skeleton supports gesture recognition by converting finger-joint geometry into finger states and mapping those states to predefined gestures.
- Gesture recognition: Finger states are determined from accumulated joint angles.Each finger is classified, for example, as bent or straight.
- Gesture recognition: Sets of finger states are mapped to predefined gestures for basic static gesture recognition.The paper describes this technique as straightforward and effective with reasonable quality.
7. Conclusion
MediaPipe Hands is an end-to-end, real-time hand-tracking pipeline for multiple platforms that predicts 2.5D landmarks without specialized hardware. The pipeline and its examples support gesture-control and AR/VR applications.
- Conclusion: MediaPipe Hands achieves real-time hand tracking on multiple platforms without specialized hardware.The pipeline predicts 2.5D landmarks and is designed for deployment on commodity devices.
- Conclusion: The pipeline is open sourced to support gesture control and creative AR/VR applications.The paper presents MediaPipe Hands as a resource for researchers and engineers.
- Conclusion: Figure 5 depicts hand-landmark controls being output when the hand-detection model is triggered.The caption attributes this behavior to MediaPipe synchronization building blocks.
- Conclusion: Figure 6 shows screenshots of real-time gesture recognition with gesture semantics rendered above the images.The figure provides an application example of the tracking pipeline.