Source-linked AI summary
AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time
Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, Cewu Lu
TL;DR
Whole-body multi-person pose estimation and tracking requires accurate localization of body, face, hand, and foot keypoints while remaining efficient. AlphaPose combines symmetric integral regression, proposal processing, training-data augmentation, and pose-aware identity embedding in a unified real-time framework. The paper reports superior efficacy and efficiency, introduces a 136-keypoint dataset, and releases an optimized library, while noting boundaries related to bottom-up scale vulnerability and gradient-analysis assumptions.
Problem
Whole-body pose estimation is challenging because fine hand and face keypoints are vulnerable to heatmap quantization error and available training data are limited.
Method
AlphaPose combines symmetric integral keypoint regression, parametric pose NMS, pose-aware identity embedding, part-guided proposal generation, and multi-domain knowledge distillation.
Results
AlphaPose is presented as a unified real-time framework with superior performance in efficacy and efficiency, supported by a new dataset containing 136 keypoints per person.
Takeaways & Limitations
The released framework, model, source code, and dataset provide a real-time system and benchmark for whole-body pose estimation and tracking.
Takeaways & Limitations
Bottom-up approaches can be vulnerable to small local regions and scale variation for small persons, while the gradient analysis assumes a smooth network gradient.
Abstract
from arXiv · showhide
Accurate whole-body multi-person pose estimation and tracking is an important yet challenging topic in computer vision. To capture the subtle actions of humans for complex behavior analysis, whole-body pose estimation including the face, body, hand and foot is essential over conventional body-only pose estimation. In this paper, we present AlphaPose, a system that can perform accurate whole-body pose estimation and tracking jointly while running in realtime. To this end, we propose several new techniques: Symmetric Integral Keypoint Regression (SIKR) for fast and fine localization, Parametric Pose Non-Maximum-Suppression (P-NMS) for eliminating redundant human detections and Pose Aware Identity Embedding for jointly pose estimation and tracking. During training, we resort to Part-Guided Proposal Generator (PGPG) and multi-domain knowledge distillation to further improve the accuracy. Our method is able to localize whole-body keypoints accurately and tracks humans simultaneously given inaccurate bounding boxes and redundant detections. We show a significant improvement over current state-of-the-art methods in both speed and accuracy on COCO-wholebody, COCO, PoseTrack, and our proposed Halpe-FullBody pose estimation dataset. Our model, source codes and dataset are made publicly available at https://github.com/MVIG-SJTU/AlphaPose.
1 INTRODUCTION
AlphaPose addresses whole-body multi-person pose estimation and tracking by combining fine keypoint localization, robust proposal handling, and unified pose-aware tracking in a real-time top-down framework.
- 1 INTRODUCTION: Top-down methods separate detection from pose estimation, so detector failures cannot be recovered and strong detectors slow inference.AlphaPose addresses redundant detections with parametric pose NMS and uses a concurrent pipeline for real-time operation.
- 1 INTRODUCTION: Whole-body multi-person pose estimation is challenging because face, hand, and foot keypoints require finer localization than conventional body-only pose estimation.Heatmap discretization can miss fine-level hand and face positions, producing quantization error.
- 1 INTRODUCTION: Symmetric integral keypoint regression localizes keypoints across scales while eliminating heatmap quantization error without the computational expense of additional hand and face subnetworks or enlarged feature maps.The method is designed to retain heatmap-level accuracy while improving fine-level localization.
- 1 INTRODUCTION: The paper introduces Halpe, a 136-point whole-body benchmark, and combines multi-domain knowledge distillation with a part-guided proposal generator to address limited and mismatched training data.PGPG simulates detector bounding boxes for different poses, augmenting training data for detected humans.
- 1 INTRODUCTION: Pose-aware identity embedding adds person identification to the pose estimator, enabling simultaneous human pose estimation and tracking within the top-down framework.Pose-guided region attention supports human identification, while the unified design targets real-time pose estimation and tracking.
- 1 INTRODUCTION: AlphaPose is presented as a unified real-time framework with accurate performance, a public whole-body dataset, and an optimized library.The reported release includes the model, source code, and dataset.
2 RELATED WORK
Prior work spans top-down, bottom-up, and one-stage approaches to multi-person pose estimation, while whole-body localization and online tracking introduce additional efficiency and precision challenges. AlphaPose follows a top-down design but adds a concurrent pipeline, integral regression, pose-guided re-identification, and multi-stage information merging.
- Multi-Person Pose Estimation: Bottom-up methods detect body parts and group them into skeletons, but local part detectors remain vulnerable to scale variation, especially for small people.This vulnerability is identified as a limitation of bottom-up approaches.
- Multi-Person Pose Estimation: AlphaPose accelerates top-down inference with a multi-stage concurrent pipeline and improves localization using symmetric integral regression.The framework also uses pose-guided re-identification and merges boxes, poses, and re-ID features for identity matching.
- Multi-Person Pose Estimation: One-stage approaches jointly locate people and joints to improve efficiency, but their high-semantic center and displacement representations make learning difficult.The cited approaches avoid separate grouping or precomputed human boxes but do not match top-down precision in the described comparison.
- Whole-Body Keypoint Localization: Whole-body methods estimate body, face, hand, and foot keypoints, yet heatmap quantization and limited output resolution reduce fine-level face and hand accuracy.Existing designs also include cascaded networks or extra computation, while AlphaPose argues that soft-argmax is more suitable for whole-body localization.
- Multi-Person Pose Tracking: Prior pose-tracking methods use graphs, video-level inputs, or designed matching mechanisms, whereas AlphaPose extracts pose-guided re-ID features and combines boxes, poses, and features.The cited contribution targets online multi-person pose tracking within the top-down framework.
3 WHOLE-BODY MULTI PERSON POSE ESTIMA-
AlphaPose addresses whole-body pose-estimation challenges through symmetric integral regression, improved confidence normalization, proposal augmentation, and parametric pose NMS. These components target fine localization, reliable joint scoring, detector-distribution mismatch, and redundant detections.
- Symmetric Integral Regression: Symmetric integral regression uses amplitude-symmetric gradients to improve learning efficiency and preserve translation invariance.The proposed gradient distribution is centered at predicted joint locations, addressing position-dependent gradient amplitudes.
- Symmetric Integral Regression: 4-times smaller Lipschitz constant than original integral regression is obtained when Agrad = W/8.The analysis indicates smoother gradients and easier optimization under this setting.
- Heatmap Normalization: Two-step heatmap normalization separates joint-confidence prediction from integral localization.Element-wise sigmoid normalization produces confidence values less affected by joint size, while global normalization produces the probability heatmap for location estimation.
- Proposal Generation: Part-guided proposal generation augments training boxes using body-part-specific detector-offset distributions.This addresses distribution differences between detected proposals and ground-truth boxes for full-body, face, and hand regions.
- Pose NMS: Parametric pose NMS removes redundant pose estimates using a data-driven pose distance with learned parameters.The method combines keypoint and heatmap similarity, with parameters determined from data rather than manually fixed.
4 MULTI PERSON POSE TRACKING
AlphaPose performs online multi-person tracking by jointly estimating pose and identity features, then matching proposals across frames with pose, appearance, and location cues. Pose-guided attention improves identity features, while multi-stage matching handles difficult associations and new tracklets.
- Identity Embedding: A person re-ID branch attached to the pose estimator jointly predicts human pose and identity features.The system uses pose-guided attention to enhance identity features before proposal matching.
- Identity Embedding: Pose-guided attention converts pose heatmaps into an attention map that weights the re-ID feature map.This focuses identity representation on regions where the human body is concentrated.
- Identity Embedding: The identity embedding is encoded as a 128-dimensional vector for human-proposal association.The embedding is produced by a fully connected layer after attention-weighted feature extraction.
- Multi-Stage Identity Matching: Multi-Stage Identity Matching first compares identity embeddings against the tracking pool and then uses position and normalized pose shape constraints.A Kalman filter smooths current detection features before matching.
- Multi-Stage Identity Matching: 0.7 is used as the identity-embedding threshold during the first matching stage.The threshold follows the cited prior setting.
- Multi-Stage Identity Matching: Unmatched proposals receive new tracklet identities after repeated lower-threshold matching fails.The second stage lowers the threshold to match tracklets that are less similar to previous frames.
5 ALPHAPOSE
AlphaPose is presented as a modular whole-body pose-estimation and tracking system with a parallelized inference pipeline, configurable detectors and estimators, and the Halpe-FullBody dataset. Its FastPose estimator uses ResNet, DUC upsampling, and a heatmap-generating convolution.
- System: AlphaPose is introduced as a jointly whole-body pose-estimation and tracking system.The implementation and source code are publicly available.
- Pipeline: A five-stage multiprocessing pipeline runs modules in parallel to enable real-time application.The stages cover loading, detection, transformation, pose estimation, and post-processing.
- System: The framework supports interchangeable human detectors and pose estimators.The current implementation uses off-the-shelf YOLOV3 and EfficientDet detectors trained on COCO.
- Pose Estimator: FastPose uses ResNet features, three DUC modules for upsampling, and a 1 × 1 convolution to generate heatmaps.A deformable-convolution variant, FastPose-DCN, further improves feature extraction.
6 DATASETS AND EVALUATIONS
The evaluation uses Halpe-FullBody and COCO-WholeBody for whole-body pose estimation, alongside COCO and PoseTrack resources for broader evaluation. Halpe-FullBody contributes 136-keypoint annotations per person and 50K training instances across body, feet, hands, and face.
- Halpe-FullBody: Halpe-FullBody annotates 136 keypoints per person: 20 body, 6 feet, 42 hands, and 68 face keypoints.
- Benchmarks: COCO-WholeBody is evaluated alongside Halpe-FullBody as a whole-body pose benchmark, with results reported in dedicated comparison tables.
- Halpe-FullBody: Halpe-FullBody contains 50K training instances and 5K testing images, using HICO-DET training images and COCO validation images.
- Keypoint definitions: The lower-jaw annotations use visible keypoints shared by two popular face-keypoint definitions, supporting compatibility between 2D and 3D conventions.
- PoseTrack: PoseTrack provides multi-person pose estimation and tracking data with more than 1,356 video sequences and 17 annotated keypoints per person.
- Supplemental datasets: Supplemental 300Wface, FreiHand, and InterHand datasets provide additional face and hand samples for improving model generalization.
6.2 Evaluation Metrics and Tools
The evaluations use whole-body and body-pose benchmarks with task-specific metrics, while reporting comparisons across accuracy, efficiency, and qualitative performance. Training uses PyTorch with a 270-epoch schedule and an additional re-identification fine-tuning phase.
- Pose metrics: Whole-body pose evaluation extends COCO Object Keypoint Similarity and reports AP 0.5:0.95:0.05, including separate body, foot, face, and hand results.
- Tracking metrics: PoseTrack tracking evaluation reports MOTA, MOTP, precision, and recall using the official poseval tool.
- Training setup: Training uses batch size 32 for 270 epochs, followed by ten epochs of re-identification fine-tuning after freezing the backbone.
- Full-body results: FastPose50-si yields 2.4 mAP, or 5.7% relatively, higher than its heatmap-based counterpart on Halpe-FullBody.
- Full-body results: On COCO-WholeBody, FastPose with symmetric integral localization outperforms previous state-of-the-art methods, especially for face and hand keypoints, while using a smaller input size.
- Body-pose results: For COCO body pose, FastPose matches state-of-the-art heatmap performance with a smaller input size and weaker detector, while regression models achieve state-of-the-art performance at the lowest GFLOPS.
6.6 Ablation Studies for Pose Estimation
Ablations assess heatmap normalization, symmetric integral regression, pose NMS, and proposal generation across full-body and body-pose settings. The experiments also compare the system’s tracking performance and efficiency against alternatives.
- Heatmap normalization: Using one-step soft-max normalization instead of two-step normalization decreases performance by 6 mAP on Halpe-FullBody and 2.4 mAP on COCO.
- SIKR module: The symmetric integral function greatly outperforms original integral regression in both full-body and conventional body-pose estimation.
- Pose NMS module: Removing Pose-NMS decreases performance by 0.4 mAP on Halpe-FullBody and 1.3 mAP on COCO because redundant poses are predicted for a single person.
- Tracking evaluation: PoseTrack evaluation shows AlphaPose outperforms most compared methods in mAP and MOTA while maintaining near-real-time speed.
- Tracking evaluation: Some methods achieve higher accuracy on PoseTrack, but use very high input and output resolutions that consume substantial memory and computation.
6.8 Ablation Studies for Pose Tracking
Tracking ablations examine pose-guided attention and the information supplied to the tracking network. The results indicate that focusing re-identification features on the target person and improving detector or pose-estimator quality benefits tracking.
- PGA module: Removing the PGA module decreases PoseTrack tracking performance, because background regions in oversized detection boxes introduce useless identity features.
- PGA module: PGA concentrates feature responses on the target person, including when two people are close together.
- MSIM ablation: Replacing the human detector and pose estimator with more accurate networks further improves tracking performance.
7 FULL BODY POSE TRACKING
AlphaPose jointly performs full-body pose estimation and tracking in crowded scenes, maintaining accuracy across human-size variation and re-identification after occlusion.
- Weakly supervised training uses both PoseTrack and Halpe-FullBody datasets for the tracking algorithm.
- Qualitative results show high accuracy for full-body pose estimation and tracking in heavily crowded scenes.
- The method remains insensitive to human-size variation during full-body pose tracking.
- After occlusion and reappearance, the method can preserve the correct identity of a person.
8 LIBRARY ANALYSIS
Compared with other open-source libraries, AlphaPose achieves high speed-accuracy performance for pose estimation and competitive pose tracking, supported by parallel processing.
- AlphaPose attains the highest accuracy and efficiency among compared libraries for whole-body and body-only pose estimation.
- Although top-down runtime increases with scene population, the parallel processing pipeline substantially reduces this deficiency.
- AlphaPose is more efficient than OpenPose when fewer than 20 persons are present.
- Its pose tracking achieves on-par performance with the state-of-the-art library.
9 CONCLUSION
The paper presents a unified realtime framework for multi-person full-body pose estimation and tracking, alongside a dataset and optimized library to support research and use.
- AlphaPose is introduced as a unified realtime framework for multi-person full-body pose estimation and tracking.
- The paper reports superior performance in both efficacy and efficiency using several novel techniques.
- The proposed Halpe-FullBody dataset annotates 136 full-body keypoints for each person.
- An optimized standard library is provided to facilitate easy usage and support the research community.
- Future library extensions are planned to include 3D keypoints and mesh.