Source-linked AI summary
Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose
Daniil Osokin
TL;DR
Multi-person pose estimation needs compact, efficient methods for real-time use on edge devices. The paper optimizes OpenPose with a lightweight network and post-processing, achieving real-time CPU performance while closely matching baseline accuracy.
Problem
The paper addresses limited compact and efficient pose estimation methods for real-time edge-device use.
Method
The method optimizes OpenPose with a lightweight network design, reduced refinement stages, and optimized post-processing.
Results
28 fps on an Intel NUC and 26 fps on a conventional CPU are reported, with less than 1% AP drop relative to the baseline.
Takeaways & Limitations
The solution runs in real time on CPUs while closely matching the baseline 2-stage network’s accuracy.
Abstract
from arXiv · showhide
In this work we adapt multi-person pose estimation architecture to use it on edge devices. We follow the bottom-up approach from OpenPose, the winner of COCO 2016 Keypoints Challenge, because of its decent quality and robustness to number of people inside the frame. With proposed network design and optimized post-processing code the full solution runs at 28 frames per second (fps) on Intel$\unicode{xAE}$ NUC 6i7KYB mini PC and 26 fps on Core$^{TM}$ i7-6850K CPU. The network model has 4.1M parameters and 9 billions floating-point operations (GFLOPs) complexity, which is just ~15% of the baseline 2-stage OpenPose with almost the same quality. The code and model are available as a part of Intel$\unicode{xAE}$ OpenVINO$^{TM}$ Toolkit.
1 Introduction
Multi-person pose estimation predicts a keypoint skeleton for every person and supports domains such as action recognition, motion capture, and sports. The paper addresses limited research on compact, efficient methods by optimizing OpenPose for edge-device performance while retaining accuracy.
- Multi-person pose estimation predicts a skeleton of keypoints for every person in an image.Keypoints include ankles, knees, hips, elbows, and other joints.
- CNNs greatly improved human pose estimation accuracy, but compact and efficient methods remain underexplored.Prior reported implementations included 10 fps on a mobile phone and 4.2 fps on a quad-core CPU.
- The authors optimize OpenPose using modern CNN design techniques for pose estimation.
- 28 fps is achieved on an Intel NUC mini PC with a 45-watt CPU TDP.
- 26 fps is achieved on a conventional CPU without a graphics card.
- Less than 1% AP drop separates the optimized version from the baseline.The comparison reports Average Precision (AP).
2 Related Work
Multi-person pose estimation can use top-down or bottom-up processing. Top-down speed depends strongly on the number of people, while bottom-up methods detect keypoints once and group them into person instances.
- Top-down estimation detects people first and runs pose estimation separately for each detected person.Its inference speed strongly depends on the number of people in the image.
- Bottom-up estimation detects all keypoints first and then groups them by human instance.This approach is usually faster because pose estimation is not rerun for each person.
- A bottom-up method reported 23 fps for three people on a GTX 1080 Ti, degrading to 15 fps for 20 people.The authors base their work on OpenPose, whose inference time is described as almost invariant to the number of people.
3 Analysis of the Original OpenPose
The original OpenPose pipeline predicts heatmaps and part affinity fields, then upsamples, extracts, and groups keypoints into person instances. Its multi-stage design provides refinement but creates complexity and post-processing costs, motivating stage reduction and optimization.
- Inference Pipeline: OpenPose has two pipeline parts: neural-network inference and grouping keypoints into person instances.
- Inference Pipeline: Network inference outputs downsampled keypoint heatmaps and pairwise part affinity fields.The output is downsampled 8 times.
- Inference Pipeline: Grouping upsamples tensors, extracts keypoints from heatmap peaks, and groups them by person.
- Inference Pipeline: The network estimates 18 keypoint types and groups them through 19 predefined keypoint pairs.Examples include left elbow–left wrist and right hip–right knee.
- Original Architecture: The original architecture uses a VGG-19 feature extractor followed by initial estimation and five refinement stages.Each stage has parallel heatmap and part-affinity-field branches.
- Complexity Analysis: Keeping only the initial stage and one refinement stage is motivated by diminishing improvement per GFLOP in later stages.
- Complexity Analysis: Post-processing profiling separates feature-map resizing, keypoint extraction, grouping, and total runtime.The implementation was written in C++ with OpenCV, and optimization focused beyond the lightweight grouping operation.
4 Optimization
The optimization reduces network computation through lightweight backbones, shared prediction branches, efficient refinement blocks, and streamlined post-processing while preserving accuracy.
- Stage reduction: ~1% AP improvement follows retraining with additional stages while using only the initial and first refinement stages.The extra stages provide a regularizing effect, although they are not used during inference.
- Lightweight backbone: Dilated MobileNet v1 preserves receptive field and spatial resolution while replacing the original VGG-based feature extractor.Removing a stride and adding dilation enables use of deeper MobileNet layers without the naive design’s accuracy drop.
- Shared prediction: A single prediction branch shares computation between heatmaps and part affinity fields, retaining separate final layers for the two outputs.The same sharing strategy is applied in both the initial and refinement stages.
- Lightweight refinement: Three convolutions with 1x1, 3x3, and dilated 3x3 kernels replace each 7x7 convolution while residual connections preserve the deeper block’s optimization.The block preserves the initial receptive field and captures long-range spatial dependencies.
- Lightweight refinement: ∼2.5 times less complexity is achieved than with a 7x7 convolution through the redesigned refinement block and additional depthwise separable convolutions.The replacement also reduces channels from 256 to 128 in the conv4_3 block.
- Fast post-processing: Upsampling remains necessary for accuracy, but factor 8 matches resizing to input size, while factor 4 was used for demonstration.Grouping directly on network output caused a significant accuracy drop.
- Evaluation: Final performance was measured with OpenVINO on an integrated GPU and a 6-core CPU for video containing more than 20 estimated poses.The network input was 456x256 with a 16:9 aspect ratio; Table 6 reports the final inference and post-processing rates.
5 Conclusion
The work targets real-time multi-person pose estimation on edge devices by optimizing OpenPose’s network and post-processing. The resulting solution substantially improves the accuracy-to-complexity ratio while retaining near-baseline accuracy and real-time CPU operation.
- Conclusion: The proposed OpenPose-based design increases the accuracy versus network complexity ratio by more than 6.5 times.This improvement uses a dilated MobileNet v1 extractor and a lightweight residual refinement stage.
- Conclusion: The full solution runs in real time on a usual CPU and an NUC mini PC while closely matching the baseline 2-stage network’s accuracy.Quantization, pruning, and knowledge distillation are identified as possible future improvements.