Source-linked AI summary
Recurrent Human Pose Estimation
Vasileios Belagiannis, Andrew Zisserman
TL;DR
2D human pose estimation requires modeling keypoint appearance and body configuration, including under occlusion. The paper proposes an end-to-end recurrent ConvNet with heatmap and auxiliary body-part supervision, trained from scratch. It achieves performance comparable to the state of the art on two challenging datasets without an explicit configuration model, and regressed heatmaps may support occlusion prediction.
Problem
Human pose estimation must handle difficult body configurations and occlusion while combining local keypoint evidence with configuration context.
Method
The paper combines feed-forward and recurrent modules that iteratively regress keypoint heatmaps, with auxiliary body-part losses and end-to-end training from scratch.
Results
The model achieves performance comparable to the state of the art on two challenging pose-estimation datasets.
Takeaways & Limitations
A simple recurrent model can capture pose context iteratively without an explicit graphical-model stage, while regressed heatmaps may help predict keypoint occlusion.
Abstract
from arXiv · showhide
We propose a novel ConvNet model for predicting 2D human body poses in an image. The model regresses a heatmap representation for each body keypoint, and is able to learn and represent both the part appearances and the context of the part configuration. We make the following three contributions: (i) an architecture combining a feed forward module with a recurrent module, where the recurrent module can be run iteratively to improve the performance, (ii) the model can be trained end-to-end and from scratch, with auxiliary losses incorporated to improve performance, (iii) we investigate whether keypoint visibility can also be predicted. The model is evaluated on two benchmark datasets. The result is a simple architecture that achieves performance on par with the state of the art, but without the complexity of a graphical model stage (or layers).
I. INTRODUCTION
The paper introduces a simple recurrent ConvNet for 2D pose estimation that combines part appearance with configuration context and achieves state-of-the-art performance without an explicit configuration model.
- The model combines a feed-forward module with a recurrent module that can run iteratively to increase the effective receptive field and improve performance.It is trained end-to-end, with auxiliary losses and an investigation of occlusion prediction.
- Keypoint heatmaps represent joints, while body-part heatmaps provide supplementary supervision for limb structure.The final keypoint prediction is produced from the keypoint heatmaps; body-part heatmaps serve as an auxiliary task.
- The model borrows fusion layers and iterative updates from prior work, applying fusion repeatedly as a recurrent network while avoiding staged training and input rendering.
- The recurrent model reaches state-of-the-art performance on standard benchmarks without an explicit configuration model or complicated network architecture.
- Iterative heatmap predictions progressively suppress false positive detections during recurrent processing.The behavior is visualized for ankle and wrist keypoints on MPII Human Pose.
A. Related Work
Related work moves from graphical models and direct coordinate regression toward heatmap-based ConvNets that capture broader context, while recurrent updates offer a parameter-efficient alternative.
- Earlier pose estimators commonly used tree-structured graphical models or poselets to encode configuration constraints.
- Heatmap regression represents each keypoint spatially and can handle multiple instances and uncertainty through multiple modes.
- Larger kernels and deeper models improve context capture but increase parameters and complicate optimization.
- Carreira et al. iteratively switch between direct pose coordinates and heatmap representations, whereas the present model uses heatmaps throughout its recurrent module.
II. RECURRENT HUMAN POSE MODEL ARCHITECTURE
The architecture separates initial part detection from recurrent context integration: a feed-forward module predicts heatmaps once, and a recurrent module updates contextual predictions iteratively.
- The model takes an image and regresses one heatmap per keypoint, with each keypoint location obtained as the heatmap mode.
- The feed-forward module mainly detects individual parts, while the recurrent module incorporates context from the configuration of other parts.
- Both modules produce heatmaps and can be trained with auxiliary losses, while the complete network is trainable end-to-end.
- The architecture contains seven layers, with the recurrent model introduced at Layers 6 and 7.The figure depicts two recurrent iterations.
A. Feed-forward Module
The feed-forward design uses compact early convolutions, larger filters for body structure, and recurrent fusion of intermediate features to improve contextual heatmap predictions.
- A. Feed-forward Module: Initial layers use 3 × 3 filters and nonlinear activations, with only two pooling operations to preserve heatmap resolution.
- A. Feed-forward Module: Larger filters in Layers 4–6 learn body structure, while 1 × 1 filters and skip concatenation form the fusion-layer inputs.
- B. Recurrent Module: The recurrent module combines fixed Layer 3 features with updated Layer 7 features at each iteration to learn context and refine final heatmaps.The network is trained end-to-end without blending predictions with the input.
- C. Body Part Heatmaps as Supplementary Supervision: Body-part heatmaps are constructed from keypoint pairs and model limbs as supplementary supervision.Their midpoint defines the Gaussian center, while variance depends on the Euclidean distance between the keypoints.
D. Target Heatmaps and Loss Function
The model trains on Gaussian target heatmaps for keypoints and uses mean squared error at both feed-forward and recurrent outputs, with auxiliary losses added across recurrent iterations.
- Target Heatmaps: Ground-truth keypoint heatmaps are synthesized by placing a fixed-variance Gaussian at each annotated keypoint.The loss measures squared pixel-wise differences between predicted and synthesized heatmaps.
- Loss Function: The same mean-squared-error loss is applied to both the feed-forward module and the recurrent module.Keypoint and body-part heatmaps are equally weighted at every loss layer.
- Auxiliary Losses: Auxiliary losses are attached to the feed-forward output and to recurrent iterations before the last one to improve optimization and increase backpropagated gradient magnitude.With two recurrent iterations, Layer 8D supplies the actual prediction while Layers 8A–8C serve as auxiliary tasks.
- Objective: The cost function compares synthesized heatmaps with ConvNet outputs parameterized by θ and indexed by recurrent iteration.The supplied equation defines this heatmap regression objective over S training samples.
E. Occluded Keypoints
The section examines occlusion as a challenge for pose estimation and tests how recurrent context affects visibility-related predictions under multiple training treatments.
- Motivation: Occlusion is challenging because body parts may be hidden by the body itself or by other entities.The discussion covers both self-occlusion and external occlusion.
- Motivation: Recurrent context can infer an occluded keypoint from the configuration of other body keypoints, but may predict it despite its invisibility.A wrist can produce a heatmap response from visibility or contextual configuration.
- Training Scenarios: The network is trained under scenarios that ignore occluded points, include them, or treat them as background and penalize their heatmap responses.These alternatives respectively vary whether occluded keypoints and body parts contribute targets, data, or penalties.
- Implementation: The model uses 248 × 248 RGB inputs and produces 62 × 62 heatmaps, with rotation, scaling, flipping, and cropping augmentation.The input is channel-wise mean-normalized.
- Implementation: Training from scratch uses a learning rate decreasing from 10^-5 to 10^-6, no more than 40 epochs, momentum 0.95, and batch size 20.Batch normalization is used in nearly all convolutional layers.
- Assumption: The experiments assume one active individual when heatmaps contain multiple people, ignoring other individuals during backpropagation.The network therefore learns to predict a single body configuration.
IV. EXPERIMENTS
The experiments evaluate model components, occlusion-related visibility prediction, and comparisons with related methods on the MPII Human Pose and LSP datasets.
- Datasets: The evaluation uses MPII Human Pose and LSP, with single-human pose estimation protocols and keypoint annotations for both datasets.MPII provides 16 keypoints and LSP provides 14.
- Qualitative Results: The recurrent-module results include heatmaps visualized after each iteration on LSP and pose predictions from two iterations on MPII Human Pose.These figures illustrate iterative recurrent predictions across the two benchmarks.
- Evaluation Design: The evaluation is divided into component analysis, occlusion evaluation, and comparison with related methods.The occlusion evaluation examines whether the model can predict keypoint visibility.
- Metric: PCKh is the main metric, counting a keypoint as correctly localized when its error is below 50% of head length.This threshold defines the reported localization criterion.
A. Component Evaluation
Component evaluation on MPII tests occlusion-label handling, recurrent iteration count, and body-part heatmaps, showing that training choices and auxiliary objectives affect performance.
- Evaluation Setup: The component study evaluates different occlusion settings, recurrent iteration counts, body-part heatmaps, cross-dataset fine-tuning, and test-time scale augmentation.Results are reported for body keypoints using PCKh on the MPII validation set.
- Occlusion Training: Including occluded points during training gives the best performance because it provides more training data.This setting forms the baseline for the proposed model.
- Recurrent Iterations: One recurrent iteration is sufficient, while adding more iterations does not improve the final result in the keypoint-only evaluation.After two iterations, further improvement is not significant.
- Body-Part Objective: Adding body-part heatmaps improves performance by helping capture additional body constraints and propagate more gradients.A one-iteration model with body parts matches a two-iteration model trained only with the keypoint objective.
B. Occlusion Prediction
The study tests whether heatmap response magnitude can predict keypoint visibility under three occlusion-labeling strategies. Penalizing occluded regions improves visibility prediction, while including occluded keypoints yields better overall pose performance because MPII evaluation ignores them.
- Visibility evaluation: The experiment infers visibility from maximum heatmap responses, treating visible keypoints as positives and occluded keypoints as negatives.Precision-recall curves are computed on the MPII Human Pose validation set, which provides occlusion labels.
- Training scenarios: The three training scenarios ignore occluded labels, include them as positive training data, or treat occluded regions as background.The models use one recurrent iteration and include body-part heatmaps.
- Visibility results: The model penalizing occluded keypoints performs better for visibility prediction than models that ignore or include them during training.Penalized occluded regions are learned as background in practice.
- Visibility results: More than 90% average precision is achieved for visibility prediction.The paper does not compare this result with another occlusion-detection method.
- Pose-performance trade-off: The penalized model is around 3% worse overall than the model including occluded keypoints because MPII evaluation ignores occluded keypoints.Including occluded keypoints supplies additional contextual training information without an evaluation penalty.
C. Comparison with other Methods
On MPII and LSP evaluations, the recurrent model achieves performance comparable to state-of-the-art methods while using a simpler and substantially smaller architecture. Its evaluation setup includes two recurrent iterations and different training choices across the datasets.
- Evaluation setup: The model is evaluated for two recurrent iterations, using no ground-truth individual localization on LSP but rough localization on MPII.This difference defines the localization setting for the two benchmark evaluations.
- MPII Human Pose Dataset: On MPII, the model achieves performance on par with other methods under the same training and validation protocol as prior work.The evaluation uses PCKh for individual keypoints and the whole body, plus AUC.
- MPII Human Pose Dataset: The model performs better than Carreira et al.’s iterative method, which relies on a pre-trained model and staged training.The comparison is reported for the MPII Human Pose evaluation.
- Model complexity: The model has two orders of magnitude fewer parameters than Hourglass and one-third as many as Convolutional Pose Machines, while maintaining comparable evaluation performance.Model-size comparisons use convolutional parameters from the examined configurations.
- LSP Dataset: On LSP, evaluation reports PCK for individual keypoints and the whole body across MPII-trained, fine-tuned, and jointly trained models.The extended LSP training set contains 10,000 samples, and training data are combined with MPII when training from scratch.
V. CONCLUSION
The paper concludes that recurrent context improves 2D pose localization while preserving comparable state-of-the-art performance on two challenging datasets. It also finds that the predicted heatmaps can support keypoint-occlusion prediction, with further work proposed for avoiding erroneous left/right assignments.
- Conclusion: The recurrent human model captures context iteratively and improves localization performance.The conclusion attributes the improvement to iterative context capture.
- Conclusion: The model reaches performance comparable to the state of the art on two challenging human pose datasets while training from scratch.The conclusion summarizes the cross-dataset result rather than reporting a single metric.
- Conclusion: Regressed heatmaps can be useful for predicting keypoint occlusion.The conclusion presents occlusion prediction as an additional use of the heatmap outputs.
- Future work: Future work will test whether combined keypoint and body-part heatmaps can prevent erroneous left/right hand assignments.The proposed use concerns the combined heatmap shown in Fig. 1(c).