Source-linked AI summary
BlazePose: On-device Real-time Body Pose tracking
Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, Matthias Grundmann
TL;DR
BlazePose addresses the challenge of making single-person pose estimation fast enough for mobile devices, where heatmap-based models can be too large. It combines heatmap and offset supervision with coordinate regression, removing those outputs at inference while retaining a lightweight tracker. The system uses 33 keypoints and is designed for real-time applications, with reported advantages in speed and Yoga/Fitness performance over OpenPose.
Problem
Heatmap-based pose estimators can be too large for real-time single-person inference on mobile devices, despite the importance of pose tracking for applications such as fitness and sign language recognition.
Method
BlazePose combines a detector-tracker pipeline, a 33-point topology, and heatmap-supervised coordinate regression with heatmap and offset outputs removed before inference.
Results
BlazePose Full outperforms OpenPose on Yoga/Fitness use cases and runs 25–75 times faster on a mid-tier phone CPU than OpenPose on a 20-core desktop CPU, depending on requested quality.
Takeaways & Limitations
The model supports near-real-time mobile use cases including Sign Language, Yoga/Fitness tracking, and AR, while its topology remains compatible with related hand and facial models.
Takeaways & Limitations
The person detector assumes that the person's head is always visible for the single-person use case.
Abstract
from arXiv · showhide
We present BlazePose, a lightweight convolutional neural network architecture for human pose estimation that is tailored for real-time inference on mobile devices. During inference, the network produces 33 body keypoints for a single person and runs at over 30 frames per second on a Pixel 2 phone. This makes it particularly suited to real-time use cases like fitness tracking and sign language recognition. Our main contributions include a novel body pose tracking solution and a lightweight body pose estimation neural network that uses both heatmaps and regression to keypoint coordinates.
1. Introduction
BlazePose targets single-person pose estimation on mobile devices, where heatmap-based models are too large for real-time inference. It combines heatmap supervision with coordinate regression and discards the heatmap branch during inference to reduce computation.
- Single-person pose estimation is difficult because poses vary widely, bodies have many degrees of freedom, and joints may be occluded.
- Heatmap-based methods handle multiple people efficiently but create models too large for real-time single-person inference on mobile phones.
- Regression approaches are cheaper and more scalable but often predict mean coordinates without resolving pose ambiguity.
- BlazePose predicts joint heatmaps with an encoder-decoder and then regresses directly to all joint coordinates with another encoder.
- The heatmap branch is discarded during inference, making the resulting model lightweight enough for mobile deployment.
2. Model Architecture and Pipeline Design
The system combines detector-tracker inference, face-based person alignment, a 33-point topology, and a heatmap-supervised regression network. Pose priors, occlusion augmentation, and visibility prediction support efficient tracking under varied viewing conditions.
- Inference pipeline: The detector-tracker pipeline uses a lightweight body detector followed by a tracker that predicts keypoints, person presence, and the next region of interest.When no person is detected, the detector runs again on the next frame.
- Person detector: NMS can fail for articulated human poses because multiple ambiguous bounding boxes may satisfy its IoU threshold.
- Person detector: The person detector uses the face as a proxy for the torso and assumes the head is always visible in the single-person use case.The face provides a strong position signal because it has high-contrast features and fewer appearance variations.
- Topology: The topology contains 33 body points, combining the keypoints used by BlazeFace, BlazePalm, and COCO for dataset and network consistency.
- Topology: The topology retains only minimally sufficient face, hand, and foot keypoints to estimate region-of-interest rotation, scale, and position.
- Neural network architecture: The model uses heatmap and offset losses only during training, then removes those output layers before inference while a regression encoder predicts coordinates.Gradient-stopping connections separate regression features from heatmap-trained features and improve coordinate regression accuracy.
- Alignment and occlusions augmentation: Limiting augmentation ranges for angle, scale, and translation supplies a pose prior that reduces network capacity and computational and energy requirements.
- Alignment and occlusions augmentation: The system aligns the hips at the image center, rotates the mid-hip-to-mid-shoulder line vertically, and scales the body to fit a square input.It also applies 10% scale and shift augmentations for inter-frame movement.
3. Experiments
BlazePose was evaluated against OpenPose on in-house AR and yoga/fitness datasets using PCK@0.2. It was slightly worse on AR, but BlazePose Full performed better on yoga/fitness and ran substantially faster on a mid-tier phone CPU.
- Evaluation setup: PCK@0.2 measured whether 2D keypoint error was below 20% of the person’s torso size on two manually annotated 1000-image datasets.The AR dataset contained varied poses in the wild, while the second contained yoga/fitness poses; both used the 17-point MS Coco topology.
- Evaluation setup: 97.2 PCK@0.2 was the average human re-annotation baseline on the AR dataset.Two annotators independently re-annotated the AR dataset to verify the human baseline.
- Results: 25–75 times faster was BlazePose than OpenPose, comparing a single mid-tier phone CPU with a 20-core desktop CPU.The speed difference depended on the requested quality.
- Results: BlazePose Full slightly underperformed OpenPose on the AR dataset but outperformed it on yoga/fitness use cases.The compared BlazePose models included Full at 6.9 MFlop and 3.5M parameters and Lite at 2.7 MFlop and 1.3M parameters.
4. Applications
BlazePose is an on-device single-person pose model designed for performance-demanding applications. Its topology and architecture support practical extensions, including more keypoints, 3D output, and additional keypoint attributes.
- Applications: BlazePose targets performance-demanding applications including Sign Language, Yoga/Fitness tracking, and augmented reality.The model operates near real time on a mobile CPU and can reach superreal-time latency on a mobile GPU.
- Applications: The model’s 33-keypoint topology is consistent with BlazeFace and BlazePalm, supporting use as a backbone for subsequent hand-pose and facial-geometry models.This topology consistency links the pose model with related inference networks.
- Extensions: The approach scales natively to more keypoints, 3D support, and additional keypoint attributes without an additional full-resolution layer for each feature type.This follows from not relying on heatmaps or offset maps at inference.
Appendix A. BlazePose keypoint names
Appendix A lists BlazePose’s body keypoints, covering facial landmarks, shoulders and limbs, hands, hips, knees, and ankles.
- Face: The keypoint list begins with facial landmarks including the nose, inner/outer eyes, ears, and mouth corners.These landmarks are assigned indices 0 through 10.
- Body and limbs: The topology includes paired shoulders, elbows, wrists, hand knuckles, hips, knees, and ankles.The passage lists left and right landmarks with consecutive indices across these body regions.