Source-linked AI summary
YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss
Debapriya Maji, Soyeb Nagori, Manu Mathew, Deepak Poddar
TL;DR
Multi-person pose estimation faces scale, occlusion, and nonrigidity challenges, while existing approaches rely on staged processing or surrogate losses. YOLO-Pose jointly detects people and poses end-to-end with anchor-based grouping and OKS-oriented supervision, achieving strong COCO results without test-time augmentation.
Problem
Multi-person pose estimation is difficult under variable scale, occlusion, and nonrigidity, while conventional heatmap approaches use L1 loss that is not equivalent to maximizing OKS.
Method
YOLO-Pose is a heatmap-free YOLOv5-based framework that jointly detects person boxes and associated poses, using anchor-based grouping and OKS loss.
Results
89.8 AP50 on COCO test-dev2017 surpasses DEKR’s 89.4 AP50, while YOLO-Pose models remain competitive on mAP at similar compute.
Takeaways & Limitations
YOLO-Pose provides a joint detection-and-pose framework with inherent grouping and standard object-detection postprocessing, positioning it as an alternative to bottom-up methods.
Takeaways & Limitations
The study limits complexity to 150 GMACS and focuses on real-time models rather than pursuing higher complexity to further close the gap with top-down approaches.
Abstract
from arXiv · showhide
We introduce YOLO-pose, a novel heatmap-free approach for joint detection, and 2D multi-person pose estimation in an image based on the popular YOLO object detection framework. Existing heatmap based two-stage approaches are sub-optimal as they are not end-to-end trainable and training relies on a surrogate L1 loss that is not equivalent to maximizing the evaluation metric, i.e. Object Keypoint Similarity (OKS). Our framework allows us to train the model end-to-end and optimize the OKS metric itself. The proposed model learns to jointly detect bounding boxes for multiple persons and their corresponding 2D poses in a single forward pass and thus bringing in the best of both top-down and bottom-up approaches. Proposed approach doesn't require the postprocessing of bottom-up approaches to group detected keypoints into a skeleton as each bounding box has an associated pose, resulting in an inherent grouping of the keypoints. Unlike top-down approaches, multiple forward passes are done away with since all persons are localized along with their pose in a single inference. YOLO-pose achieves new state-of-the-art results on COCO validation (90.2% AP50) and test-dev set (90.3% AP50), surpassing all existing bottom-up approaches in a single forward pass without flip test, multi-scale testing, or any other test time augmentation. All experiments and results reported in this paper are without any test time augmentation, unlike traditional approaches that use flip-test and multi-scale testing to boost performance. Our training codes will be made publicly available at https://github.com/TexasInstruments/edgeai-yolov5 and https://github.com/TexasInstruments/edgeai-yolox
1. Introduction
YOLO-Pose frames multi-person pose estimation as joint person detection and pose localization, addressing scale variation, occlusion, and grouping challenges with a heatmap-free YOLO-based approach. Its anchor-associated poses provide inherent grouping while avoiding top-down variable runtime and bottom-up grouping postprocessing.
- Multi-person pose estimation must handle variable person counts, scale variation, occlusion, and human-body nonrigidity.
- Top-down methods scale linearly with the number of people, whereas bottom-up methods use constant-runtime heatmaps but require complex keypoint grouping.
- YOLO-Pose adapts object-detection strategies to pose estimation, including multi-scale predictions for challenges such as scale variation.
- The heatmap-free YOLOv5-based approach uses standard object-detection postprocessing and reports competitive AP with improved AP50 on COCO without non-standardized pose postprocessing.
- YOLO-Pose stores each person’s full 2D pose with a matched anchor box or point, so associated keypoints are inherently grouped.
- YOLO-Pose jointly detects people and estimates poses in one forward pass, combining constant runtime with simple postprocessing.
2. Related Work
Related work divides multi-person pose estimation into top-down and bottom-up approaches. Top-down methods estimate each detected person separately, while bottom-up methods detect identity-free keypoints and group them into person instances.
- Multi-person 2D pose estimation is commonly categorized into top-down and bottom-up approaches.
- Top-down Methods: Top-down methods first detect people with a heavy detector and then estimate 2D pose for each detected person.
- Top-down Methods: Top-down computational complexity increases linearly with the number of persons, while processing subjects at the same scale provides scale invariance.
- Bottom-up Methods: Bottom-up methods detect identity-free keypoints in one shot and use subsequent grouping to form individual person instances.
- Bottom-up Methods: Bottom-up systems use heatmaps and additional adjustment, refinement, and grouping strategies, including part affinity fields or tag values.
3. YOLO-Pose
YOLO-Pose extends a single-shot YOLO detector to predict each person's bounding box and complete 2D pose without heatmaps. It uses anchor-associated keypoints, OKS-based supervision, and standard object-detection processing for real-time multi-person estimation.
- 3. YOLO-Pose: YOLO-pose associates all keypoints of each person with anchors, avoiding heatmaps and preserving inherent grouping of each pose.Each anchor matched to a person stores the entire 2D pose along with its bounding box.
- 3.1. Overview: The model is based on YOLOv5 and uses CSP-darknet53, PANet, four detection scales, and decoupled box and keypoint heads.The backbone generates multiscale features, PANet fuses them, and each detection head branches into box and keypoint heads.
- 3.2. Prediction Representation: For each anchor, the pose head predicts 51 values for 17 keypoints, while the box head predicts six values for a single person class.Each keypoint includes a location and confidence, and the overall prediction vector is defined per anchor.
- 3. YOLO-Pose: The method targets real-time models within 150 GMACS and reports results without test-time augmentation.Flip testing and multiscale testing increase complexity, while the reported approach avoids those extra forward passes and data-processing costs.
- 3.4. Human Pose Loss Function Formulation: YOLO-Pose directly optimizes OKS for regressed keypoints instead of using the conventional L1 surrogate loss.OKS loss is scale-invariant, weights keypoints differently, and is computed separately for each keypoint before aggregation.
- 3.6. Keypoint Outside Bounding Box: The approach retains keypoints outside predicted person boxes, allowing occluded or inaccurately boxed keypoints to remain detectable.This removes the top-down constraint that pose estimation fails when person detection is incorrect.
4. Experiments
Experiments evaluate YOLO-Pose on COCO using standard OKS-based metrics, comparing accuracy, complexity, loss functions, resolutions, and quantization settings. The results show competitive performance at similar compute, strong AP50, and relatively small accuracy losses under selected quantization schemes.
- Experimental setup: The model is trained on COCO train2017 and evaluated on val2017 and test-dev2017 using AP and AR metrics at multiple OKS thresholds and object sizes.COCO contains over 200,000 images, 250,000 person instances, and 17 keypoints.
- Experimental setup: Training uses YOLOv5-like augmentation, anchor selection, and loss weighting, with random scale, translation, flipping, mosaic, and color augmentation.The model is optimized with SGD and a cosine scheduler for 300 epochs.
- COCO results: YOLO-Pose models are competitive with similar-compute methods in AP and show their strongest gains in AP50 against bottom-up and DEKR baselines.YOLOv5m6-pose exceeds DEKR models with four times higher complexity in AP50, while YOLOv5l6-Pose also improves APL over DEKR models with twice higher complexity.
- Ablation: OKS loss versus L1 loss: OKS loss significantly outperforms L1 loss on YOLOv5-s6_960, while adding scale information improves accuracy and OKS loss remains the most suitable for optimizing OKS.The authors report more stable training dynamics with the constrained OKS loss.
- Ablation: resolution: Performance gains largely saturate beyond the selected 960 resolution, while YOLOv5s6-pose performs significantly better than EfficientHRNet at lower resolutions.The low-complexity comparison covers models below 30 GMACS.
- Ablation: quantization: Quantization causes approximately a 1.2% accuracy drop in the reported mixed-precision setting, whereas 16-bit quantization reduces AP by 0.1% and leaves AP50 unchanged.The mixed-precision setting places approximately 30% of layers in 16 bits; these results use post-training quantization rather than quantization-aware training.
5. Conclusion
The paper presents an end-to-end YOLOv5-based framework for joint detection and multi-person pose estimation, reporting lower complexity than existing bottom-up approaches. It positions this approach as an initial step toward unifying object detection and human pose estimation.
- The framework jointly detects persons and estimates multi-person poses end-to-end using YOLOv5.
- The models outperform existing bottom-up approaches at significantly lower complexity.
- The approach aims to transfer advances from object detection to human pose estimation.
- Preliminary YOLOX experiments achieved promising results, with extensions to other object detection frameworks planned.