Source-linked AI summary
RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose
Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, Kai Chen
TL;DR
Industrial deployment of 2D multi-person pose estimation remains constrained by heavy models and high latency despite strong public-benchmark performance. RTMPose empirically optimizes the paradigm, architecture, training, and deployment pipeline, achieving high accuracy with real-time inference across CPU, GPU, and mobile devices.
Problem
2D pose estimation performs strongly on public benchmarks, but industrial use remains limited by heavy model parameters and high latency.
Method
RTMPose combines a top-down paradigm, CSPNeXt backbone, SimCC localization, training strategies, and jointly optimized deployment components.
Results
75.8% AP is achieved by RTMPose-m on COCO val with 90+ FPS on an Intel i7-11700 CPU and 430+ FPS on an NVIDIA GeForce GTX 1660 Ti GPU.
Takeaways & Limitations
RTMPose balances model performance and complexity and supports real-time inference on CPU, GPU, and mobile devices.
Abstract
from arXiv · showhide
Recent studies on 2D pose estimation have achieved excellent performance on public benchmarks, yet its application in the industrial community still suffers from heavy model parameters and high latency. In order to bridge this gap, we empirically explore key factors in pose estimation including paradigm, model architecture, training strategy, and deployment, and present a high-performance real-time multi-person pose estimation framework, RTMPose, based on MMPose. Our RTMPose-m achieves 75.8% AP on COCO with 90+ FPS on an Intel i7-11700 CPU and 430+ FPS on an NVIDIA GTX 1660 Ti GPU, and RTMPose-l achieves 67.0% AP on COCO-WholeBody with 130+ FPS. To further evaluate RTMPose's capability in critical real-time applications, we also report the performance after deploying on the mobile device. Our RTMPose-s achieves 72.2% AP on COCO with 70+ FPS on a Snapdragon 865 chip, outperforming existing open-source libraries. Code and models are released at https://github.com/open-mmlab/mmpose/tree/1.x/projects/rtmpose.
1. Introduction
RTMPose addresses the industrial gap in real-time multi-person pose estimation by combining empirical design choices across the pipeline with deployment-oriented optimization. Its top-down framework targets a balance of accuracy, speed, robustness, and device coverage.
- Real-time multi-person pose estimation remains difficult on devices with limited computing power despite strong benchmark progress.
- RTMPose empirically studies paradigm, backbone, localization, training strategy, and deployment to build real-time pose models.
- Top-down inference uses an off-the-shelf detector followed by individual pose estimation, remaining real time in most scenarios within 6 persons per image.
- CSPNeXt provides a deployment-friendly backbone that balances speed and accuracy, while SimCC formulates localization as classification with lower computational effort than heatmaps.
- Training strategies, skip-frame detection, pose NMS, smoothing, and multiple model sizes jointly target accuracy, latency, robustness, and application-specific trade-offs.
- 75.8% AP is achieved by RTMPose-m on COCO val with 90+ FPS on an Intel i7-11700 CPU and 430+ FPS on an NVIDIA GeForce GTX 1660 Ti GPU.
2. Related Work
Related work spans bottom-up, top-down, coordinate-classification, and transformer-based approaches to multi-person pose estimation. RTMPose’s architecture follows coordinate classification while emphasizing compact computation and deployment practicality.
- Bottom-up Approaches: Bottom-up methods detect instance-agnostic keypoints and partition them into poses, offering stable computational cost as crowd size increases.
- Bottom-up Approaches: Bottom-up methods often require large input resolutions for varied person scales, making accuracy and inference speed difficult to reconcile.
- Top-down Approaches: Top-down methods detect bounding boxes and crop each person to a uniform scale, enabling strong speed and accuracy in non-extreme scenarios.
- Coordinate Classification: SimCC formulates keypoint prediction as classification over sub-pixel bins for horizontal and vertical coordinates, reducing dependence on high-resolution heatmaps.
- Vision Transformers: Transformer-based pose methods improve representation or accuracy but can retain heatmaps, pixel tokens, or keypoint tokens that increase computation and hinder real-time inference.
- RTMPose’s architecture combines convolution, a fully connected layer, and a Gated Attention Unit before predicting horizontal and vertical coordinates as classification tasks.
3. Methodology
RTMPose builds a lightweight, deployment-friendly top-down pose estimator by refining SimCC across architecture, training, modules, and the inference pipeline.
- RTMPose follows a coordinate-classification roadmap, refitting SimCC with efficient backbones, training strategies, modules, micro designs, and pipeline optimization.
- SimCC classifies horizontal and vertical keypoint coordinates into bins, using Gaussian soft labels to encode spatial relationships during training.
- Removing SimCC upsampling layers and using CSPNext-m produces a lightweight baseline with 69.7% AP.
- UDP pre-training, EMA, flat cosine annealing, and two-stage augmentation progressively improve optimization and training performance.UDP pre-training raises accuracy from 69.7% AP to 70.3% AP; EMA and flat cosine annealing further raise it to 70.7% AP.
- Feature expansion and GAU self-attention refine keypoint representations, with the self-attention module improving performance by 0.5% AP to 71.9%.The feature dimension is set to 256, improving accuracy from 71.2% AP to 71.4% AP.
- Temperature scaling, separate horizontal and vertical σ values, larger kernels, longer training, and multi-dataset training raise performance to 75.3% AP.Using τ = 0.1 raises accuracy from 71.9% to 72.7%; separate σ raises it to 72.8%, a 7 × 7 kernel reaches 73.3% AP, and multi-dataset training reaches 75.3% AP.
4. Experiments
The experiments evaluate RTMPose across body, single-person, whole-body, and other pose datasets, while measuring complexity and deployment speed across CPU, GPU, and mobile hardware. RTMPose generally balances accuracy and efficiency, with strong performance against open-source alternatives.
- COCO: RTMPose outperforms competitors on COCO with lower complexity and strong robustness to different off-the-shelf detectors.
- COCO-SinglePerson: RTMPose achieves superior performance and efficiency on COCO-SinglePerson compared with solutions designed for single-person scenarios.The comparison includes BlazePose, MoveNet, and PaddleDetection.
- COCO-WholeBody: RTMPose-m outperforms previous open-source libraries on COCO-WholeBody with significantly lower GFLOPs.Increasing input resolution and training data yields competitive accuracy with state-of-the-art approaches.
- Other Datasets: RTMPose is further evaluated on AP-10K, CrowdPose, and MPII using ImageNet pre-training for fair comparison with baselines.Models pre-trained on combined COCO and AI Challenger data achieve higher accuracy and are reproducible with provided weights.
- Deployment: The evaluation measures inference speed across CPU, GPU, and mobile deployments using MMDeploy and specified inference backends.Mobile testing uses Snapdragon 865 with ncnn; CPU and GPU tests use ONNXRuntime and TensorRT respectively.
5. Conclusion
The paper presents RTMPose as a high-performance real-time multi-person pose estimation framework designed to balance model performance and complexity across deployment targets. Its open-source implementation is intended to support practical industrial applications and future pose-estimation research.
- RTMPose balances model performance and complexity while supporting real-time inference on CPU, GPU, and mobile devices.
- The paper empirically explores pose-estimation paradigm, model architecture, training strategy, and deployment.
- The authors aim for RTMPose and its open-source implementation to meet some industrial demand and benefit future human pose estimation research.