Source-linked AI summary
PoseFix: Model-agnostic General Human Pose Refinement Network
Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee
TL;DR
Human pose refinement is needed beyond conventional multi-stage systems that depend on specific pose estimators and careful design. PoseFix uses empirical error statistics to synthesize training poses for a model-agnostic, coarse-to-fine refiner. It outperforms conventional refinement and consistently improves various state-of-the-art estimators, while remaining usable as a post-processing add-on.
Problem
Conventional multi-stage pose refinement depends on the pose-estimation model and requires careful model design.
Method
PoseFix uses empirical pose-error distributions to synthesize training poses and trains a model-agnostic coarse-to-fine refiner on image–pose pairs.
Results
PoseFix outperforms conventional multi-stage refinement methods and consistently improves various state-of-the-art pose estimators on a common benchmark.
Takeaways & Limitations
PoseFix can refine results from any pose-estimation method without requiring that method's code or internal knowledge.
Takeaways & Limitations
Using a refinement module with the same architecture as PoseFix can be sub-optimal because conventional refinement requires careful network design under limited GPU memory.
Abstract
from arXiv · showhide
Multi-person pose estimation from a 2D image is an essential technique for human behavior understanding. In this paper, we propose a human pose refinement network that estimates a refined pose from a tuple of an input image and input pose. The pose refinement was performed mainly through an end-to-end trainable multi-stage architecture in previous methods. However, they are highly dependent on pose estimation models and require careful model design. By contrast, we propose a model-agnostic pose refinement method. According to a recent study, state-of-the-art 2D human pose estimation methods have similar error distributions. We use this error statistics as prior information to generate synthetic poses and use the synthesized poses to train our model. In the testing stage, pose estimation results of any other methods can be input to the proposed method. Moreover, the proposed model does not require code or knowledge about other methods, which allows it to be easily used in the post-processing step. We show that the proposed approach achieves better performance than the conventional multi-stage refinement models and consistently improves the performance of various state-of-the-art pose estimation methods on the commonly used benchmark. The code is available in this https URL\footnote{\url{https://github.com/mks0601/PoseFix_RELEASE}}.
1. Introduction
PoseFix addresses the dependence and careful-design requirements of conventional multi-stage pose refinement with a model-agnostic refiner trained from empirical pose-error statistics. It accepts poses from other estimation methods and reports stronger refinement performance with broad applicability.
- Conventional multi-stage refinement depends heavily on the pose-estimation model and requires careful architecture design.
- PoseFix generates synthetic poses from empirical distributions of jitter, inversion, swap, and miss errors, then trains on image–pose pairs.
- PoseFix is trained independently of the pose-estimation model, enabling model-agnostic pose refinement.
- Any pose-estimation method can provide PoseFix's input without requiring its code or internal knowledge.
- The coarse-to-fine pipeline is empirically described as crucial for successful pose refinement.
- PoseFix outperforms conventional multi-stage refinement methods and consistently improves various state-of-the-art estimators on a common benchmark.
3. Overview of the proposed model
PoseFix trains on diverse, realistic poses synthesized by applying empirically observed errors to ground-truth poses. At test time, it refines poses from other methods using image context and structured pose information.
- PoseFix processes a cropped human image and an input pose, synthesizing training poses from ground-truth poses and accepting other methods' results at test time.
- Synthetic poses are sampled from error distributions conditioned on joint type, visible-keypoint count, and image overlap.
- For each joint, d^k_j is the L2 distance that yields keypoint similarity k, with scale varying by joint type.
- The synthesis procedure models good, jitter, inversion, swap, and miss pose-error statuses.
5. Architecture and learning of PoseFix
PoseFix refines an input pose using both image context and structured pose information, progressing from coarse Gaussian representations to precise coordinates. Its design combines heatmap prediction, differentiable coordinate extraction, and losses that support both discrete localization and continuous precision.
- PoseFix estimates a refined pose from an input image and input pose, using image context and structured pose information to correct errors.
- Model behavior: Feature-map visualizations show that PoseFix first attends to reliable input-keypoint locations and subsequently localizes correct keypoints despite input errors.Feature maps and final heatmaps are reduced to one channel by channel-wise max pooling, in feedforward order.
- Coarse-to-fine estimation: The network processes poses coarse-to-fine, transforming Gaussian-blurred keypoint representations into one-hot heatmaps and then final keypoint coordinates.The coarse input pose is represented by Gaussian blobs; the finer pose is represented by heatmaps and the final output by coordinates.
- Coarse-to-fine estimation: Gaussian heatmaps are pixel-aligned with the image and retain non-zero neighborhoods that help the model consider locations around potentially erroneous input keypoints.
- Coarse-to-fine estimation: Soft-argmax converts each predicted heatmap into differentiable 2D coordinates by combining heatmap values with a spatial meshgrid.
- Losses: The integral loss combines heatmap cross-entropy with coordinate L1 loss, enforcing discrete heatmap selection while improving localization in continuous space.The heatmap loss uses spatial softmax, whereas the coordinate loss avoids quantization errors.
6. Implementation details
PoseFix is trained end-to-end with a pretrained ResNet backbone and Adam optimization, then accepts pose estimates from other methods during testing. The implementation also uses bounding boxes derived from input keypoints and test-time flip augmentation.
- Training: PoseFix is trained end-to-end with an ImageNet-pretrained ResNet backbone and Adam optimization using mini-batches of 128.The initial learning rate is 5×10^-4 and is reduced by a factor of 10 at the 90th and 120th epochs.
- Testing: During testing, pose estimates from other methods provide the input pose, and human bounding boxes are calculated from those keypoint coordinates.
- Testing: Testing uses flip augmentation, and the model is implemented with the TensorFlow deep learning framework.
7. Experiment
Experiments show that PoseFix benefits from model-agnostic training and coarse-to-fine estimation, outperforming conventional refinement and consistently improving diverse pose estimators.
- Model-agnostic pose refinement: PoseFix improves accuracy more than conventional end-to-end multi-stage refinement when trained independently of the pose estimation model.The comparison uses E2E-refine and MA-refine on the validation set.
- Model-agnostic pose refinement: Synthetic realistic input poses act as data augmentation, making PoseFix more robust to unseen poses at test time.The poses are generated from empirical error statistics rather than a single estimator's outputs.
- Performance improvement: Because training and testing are decoupled, PoseFix can function as an add-on module without requiring careful redesign of the estimator.The paper contrasts this flexibility with memory-constrained custom refinement-module design.
- Coarse-to-fine estimation: The C2F pipeline outperforms F2F and C2C, indicating that coarse input representation combined with fine-form estimation is beneficial.C2C fails to improve the input pose, whereas F2F and C2F refine it successfully.
- Coarse-to-fine estimation: Finer output representations improve performance: C2C is worst, C2F-LC is best among single-loss variants, and integrating LH and LC yields C2F.The analysis identifies fine-form estimation and combined losses as important components.
- Performance improvement: PoseFix consistently improves state-of-the-art methods across AP and OKS evaluations on validation and test-dev sets.The evaluated methods include PAFs, AE, Mask R-CNN, CPN, and Simple.
- Performance improvement: PoseFix corrects jitter as well as larger displacement errors, including inversion, miss, and swap.Error-frequency changes are analyzed when PoseFix is applied to CPN.
8. Conclusion
PoseFix is a model-agnostic pose refinement network that synthesizes training errors from empirical distributions and refines coarse input poses into finer ones. It outperforms conventional multi-stage refinement and improves other methods on a common benchmark.
- 8. Conclusion: PoseFix is a model-agnostic network that refines coarse input poses into finer poses.It is trained using synthetic errors sampled from empirical pose-error distributions.
- 8. Conclusion: PoseFix does not require code or knowledge about target pose-estimation models, enabling convenient post-processing use.
- 8. Conclusion: PoseFix outperforms conventional end-to-end multi-stage refinement modules and consistently improves other methods on a commonly used benchmark.
Refinement Network”
The supplementary material presents additional experimental results omitted from the main manuscript because of limited space.
- Refinement Network”: The supplementary material reports experiments that could not fit in the main manuscript.
1. Comparison with conventional end-to-end trainable multi-stage refinement
The comparison evaluates model-agnostic PoseFix against conventional refinement under controlled and state-of-the-art settings. PoseFix improves AP more than RefineNet by a large margin.
- 1. Comparison with conventional end-to-end trainable multi-stage refinement: The comparison matches E2E-refine and MA-refine parameter counts to evaluate model-agnostic refinement effectiveness.
- 1. Comparison with conventional end-to-end trainable multi-stage refinement: Because conventional refinement requires careful model design, simply adding a similarly structured refinement module can produce sub-optimal performance.
- 1. Comparison with conventional end-to-end trainable multi-stage refinement: PoseFix improves AP more than the state-of-the-art RefineNet by a large margin.The comparison uses GlobalNet as the pose-estimation model and contrasts RefineNet and PoseFix refinement.
2. Performance improvement of the state-ofthe-art methods by PoseFix
PoseFix improves multiple state-of-the-art pose estimators across MS COCO and PoseTrack 2018. The reported error analyses indicate improvements across all pose-error types and across variable datasets.
- 2. Performance improvement of the state-ofthe-art methods by PoseFix: PoseFix improves performance by fixing all types of pose errors for AE and Mask R-CNN on MS COCO.
- 2. Performance improvement of the state-ofthe-art methods by PoseFix: Table 3 compares AP between PoseFix and conventional RefineNet refinement on the MS COCO validation set.Parenthesized values denote AP changes from the GlobalNet input pose.
- 2. Performance improvement of the state-ofthe-art methods by PoseFix: Table 4 compares AP with state-of-the-art methods on the test-dev set, marking extra training data and ensemble results.
- 2. Performance improvement of the state-ofthe-art methods by PoseFix: PoseFix improves the input pose on PoseTrack 2018 using the Simple estimator.The experiment uses the PoseTrack 2018 validation set and supports generalization across datasets.
3. Comparison with state-of-the-art methods
PoseFix is evaluated against state-of-the-art pose estimation methods on MS COCO test-dev and against PoseRefiner on PoseTrack 2018 validation. It outperforms the compared methods and improves the input pose more than PoseRefiner.
- MS COCO test-dev comparison: PoseFix is compared with PAFs, G-RMI, AE, RMPE, Mask R-CNN, CFN, CPN, Integral, MultiPoseNet, and Simple on MS COCO test-dev.Simple provides the input pose for PoseFix; reported comparison results come from the respective papers, with the authors’ human detector used where needed.
- MS COCO test-dev comparison: PoseFix outperforms all existing methods in the MS COCO test-dev comparison.
- PoseTrack 2018 validation comparison: PoseTrack 2018 validation evaluates AP improvement when PoseFix is applied to state-of-the-art methods.
- MS COCO test-dev comparison: PoseFix improves a newly proposed state-of-the-art method when that method’s pose estimates are used as its input.
- PoseTrack 2018 validation comparison: On PoseTrack 2018 validation, PoseFix improves the input pose significantly more than PoseRefiner.Table 6 compares AP between PoseRefiner and PoseFix; parenthesized values denote AP change from the Simple input pose.
- Qualitative results: Qualitative results show refined poses from Mask R-CNN inputs and final results from Simple inputs on the MS COCO test-dev set.