MediaPipe Hands: On-device Real-time Hand Tracking

• Hand tracking supports natural interaction and communication in AR/VR applications.Source: §1 · p. 1
• The pipeline tracks multiple hands in real time on mobile devices using an efficient two-stage design.Source: §1 · p. 1
• It predicts 2.5D hand pose using only RGB input, without additional specialized hardware.Source: §1 · p. 1
• The hand-tracking pipeline is open source and supports Android, iOS, Web, and desktop PCs.Source: §1 · p. 1

2. Architecture

• The two-stage pipeline detects palms in the full image, then predicts high-fidelity 2.5D landmarks inside the cropped hand region.The detector supplies an oriented hand bounding box to the landmark model.Source: §2 · p. 1
• Reusing the previous frame’s landmark-derived bounding box avoids running the palm detector on every frame.The detector runs on the first frame or when tracking indicates that the hand has been lost.Source: §2 · p. 1

2.1. BlazePalm Detector

• The palm detector targets scale variation, occlusion, and difficult hand appearance using palm-specific boxes, scene-context features, and focal loss.Palm boxes simplify detection because palms are more rigid, can use square boxes, and reduce the number of anchors by 3∼5.Source: §2.1 · p. 2

2.2. Hand Landmark Model

• The landmark model regresses 21 coordinates and also outputs hand presence and binary handedness classifications.The 21 landmarks contain x, y, and relative depth; the hand-presence score can trigger detector-based tracking reset.Source: §2.2 · p. 2
• The landmark model learns 2D pose from real and synthetic images, while relative depth is learned only from synthetic images.The model uses a shared feature extractor with three output heads and supports lighter or heavier variants for different hardware and accuracy needs.Source: §2.2 · p. 2, Figure 3 · p. 2

3. Dataset and Annotation

• The in-the-wild dataset contains 6K diverse images, while the in-house gesture dataset contains 10K images from 30 people.The datasets complement one another through appearance diversity and coverage of physically possible hand gestures, despite their respective limitations.Source: §3 · p. 3
• The synthetic dataset expands possible hand-pose coverage and provides additional depth supervision.Synthetic images use projected ground-truth 3D joints for landmark supervision.Source: §3 · p. 3
• The palm detector is trained only on the in-the-wild dataset, whereas all datasets train the hand landmark model.Real images receive 21-landmark annotations, while synthetic images use projected ground-truth 3D joints.Source: §3 · p. 3

Dataset evaluation

• Real-world and synthetic datasets together provide the best hand landmark model results.Evaluation is conducted on real-world images, and the synthetic data also reduces visible frame-to-frame jitter.Source: §4 · p. 3

Model capacity

• The Full model provides a good trade-off between quality and speed.Increasing capacity further yields only minor quality improvements while significantly reducing speed.Source: §4 · p. 3

Dataset evaluation

• Table 2 reports results for models trained from different datasets.The comparison concerns dataset composition for the hand landmark model.Source: Table 2 · p. 3

Model capacity

• Table 3 reports hand landmark model performance characteristics.These characteristics support comparison of model quality and speed.Source: Table 3 · p. 3

MediaPipe graph

• MediaPipe represents hand tracking as a directed graph of modular Calculator components.Calculators handle model inference, media processing, and data transformations, with GPU acceleration used for operations including cropping and neural-network computation.Source: §5 · p. 4

Tracking optimization

• The palm detector runs only when needed rather than on every video frame.The current hand location is derived from previous-frame landmarks, and detection is reapplied when hand-presence confidence falls below a threshold.Source: §5 · p. 4

Gesture recognition

• Finger states are determined from accumulated joint angles.Each finger is classified, for example, as bent or straight.Source: §6 · p. 4
• Sets of finger states are mapped to predefined gestures for basic static gesture recognition.The paper describes this technique as straightforward and effective with reasonable quality.Source: §6 · p. 4

Conclusion

• MediaPipe Hands achieves real-time hand tracking on multiple platforms without specialized hardware.The pipeline predicts 2.5D landmarks and is designed for deployment on commodity devices.Source: §7 · p. 4
• The pipeline is open sourced to support gesture control and creative AR/VR applications.The paper presents MediaPipe Hands as a resource for researchers and engineers.Source: §7 · p. 4
• Figure 5 depicts hand-landmark controls being output when the hand-detection model is triggered.The caption attributes this behavior to MediaPipe synchronization building blocks.Source: Figure 5 · p. 4
• Figure 6 shows screenshots of real-time gesture recognition with gesture semantics rendered above the images.The figure provides an application example of the tracking pipeline.Source: Figure 6 · p. 4