Source-linked AI summary
Human Pose Estimation with Iterative Error Feedback
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik
TL;DR
Feedforward ConvNets represent images well but do not explicitly model the structured dependencies of outputs such as human poses. IEF adds top-down feedback by iteratively predicting corrections to an initial solution, achieving competitive MPII and LSP pose-estimation results, including without ground-truth scale annotation.
Problem
Feedforward architectures learn rich input representations but do not explicitly model structured output dependencies needed for tasks such as articulated human pose estimation.
Method
IEF concatenates the image with a rendering of the current output and repeatedly predicts corrections that update the estimate.
Results
IEF achieves PCKh-0.5 of 81.0, compared with 74.8 for direct keypoint regression and 73.4 for iterative direct prediction.
Takeaways & Limitations
IEF extends hierarchical representation learning to output spaces by shifting prediction from estimating states directly to correcting current expectations.
Takeaways & Limitations
The method has so far used Gaussian-rendered feedback and may require more powerful learned renderings for higher-dimensional outputs such as 3D pose.
Abstract
from arXiv · showhide
Hierarchical feature extractors such as Convolutional Networks (ConvNets) have achieved impressive performance on a variety of classification tasks using purely feedforward processing. Feedforward architectures can learn rich representations of the input space but do not explicitly model dependencies in the output spaces, that are quite structured for tasks such as articulated human pose estimation or object segmentation. Here we propose a framework that expands the expressive power of hierarchical feature extractors to encompass both input and output spaces, by introducing top-down feedback. Instead of directly predicting the outputs in one go, we use a self-correcting model that progressively changes an initial solution by feeding back error predictions, in a process we call Iterative Error Feedback (IEF). IEF shows excellent performance on the task of articulated pose estimation in the challenging MPII and LSP benchmarks, matching the state-of-the-art without requiring ground truth scale annotation.
1. Introduction
IEF extends feedforward ConvNets with top-down feedback to model dependencies in structured outputs such as human pose. It iteratively predicts corrections to an initial pose while learning features over the joint space of images and body configurations.
- 1. Introduction: Structured human poses encode body proportions, symmetries, joint limits, occlusion relationships, and physical connectivity.Modeling these dependencies can help localize visible keypoints and estimate occluded ones.
- 1. Introduction: IEF predicts what is wrong with a current output estimate and corrects it iteratively instead of directly predicting the target.The framework is designed as a generic approach for modeling rich structure in both input and output spaces.
- 1. Introduction: At iteration t, a ConvNet receives the image concatenated with a rendering of current keypoint positions and outputs a correction ϵt.The correction updates yt, whose rendered representation is then fed back with the image at the next iteration.
- 1. Introduction: For pose estimation, each keypoint is rendered as a fixed-width Gaussian heatmap, producing K additional channels stacked with the RGB image.These heatmaps represent current beliefs about keypoint locations in the image plane.
- 1. Introduction: The augmented input to f has dimensions H × W × (K + 3), allowing the ConvNet to learn features over images and estimated outputs jointly.The K keypoint channels are combined with the three color channels.
2. Learning
IEF learning constructs fixed intermediate correction targets because only the final ground-truth output is assumed to be available. Fixed Path Consolidation progressively adds later correction steps while training bounded, directional updates for each pose keypoint.
- 2. Learning: The model minimizes a distance between predicted and target bounded corrections over T correction steps, with T fixed or determined by a termination condition.The predicted and target corrections are compared using a loss such as quadratic distance.
- 2. Learning: FPC first trains on samples for the earliest correction step, then progressively adds later steps so early corrections receive longer optimization.This curriculum-style procedure consolidates the learned path through output space.
- 2. Learning: Fixed Path Consolidation (FPC) predefines intermediate targets by applying fixed corrections from an initial estimate toward the ground-truth output.This resolves the ambiguity of what intermediate targets should be when only the final output is given during training.
- 2.1. Learning Human Pose Estimation: Pose training renders predicted keypoint locations as heatmaps, feeds them with the image to a ConvNet, and uses the predicted corrections to refine locations iteratively.The implementation trains for T = 4 steps but uses 3 steps at test time because the fourth step had little effect on accuracy.
- 2.1. Learning Human Pose Estimation: For each keypoint, the target correction moves in the direction of the ground truth by at most L, remaining constant when far away and shrinking near the target.This turns learning into predicting a constant direction followed by slower motion near the ground truth.
3. Results
IEF was evaluated on MPII and LSP, including settings with and without ground-truth scale information. It roughly matched state of the art and substantially improved MPII performance when scale was unknown.
- IEF was tested on MPII and LSP, challenging benchmarks featuring scale variation, occlusion, interacting people, and complex sports poses.
- 81.3 PCKh was achieved on MPII when ground-truth scale information was unavailable, compared with 66.0 PCKh from the previous best result.
- IEF roughly matched the state of the art on MPII under the standard evaluation setting with ground-truth scale information.
- Person-centric PCP scores on LSP were competitive with the current state of the art of Chen and Yuille.
4. Analyzing IEF
Ablation studies examined iterative correction, bounded errors, Fixed Path Consolidation, and structured output modeling. The results favor IEF's iterative bounded-correction approach and its curriculum-style training procedure.
- Iterative v/s Direct Prediction: 81.0 PCKh-0.5 for IEF exceeded 74.8 PCKh-0.5 for direct regression to keypoint locations.
- Iterative Error Feedback v/s Iterative Direct Prediction: 81.0 PCKh-0.5 for IEF exceeded 73.4 PCKh-0.5 for iterative direct prediction of ground-truth keypoint locations.
- Importance of Fixed Path Consolidation (FPC): Without Fixed Path Consolidation, validation performance dropped by almost 10 PCKh points and several correction steps produced significant drift.
- Learning Structured Outputs: IEF models augment images with Gaussian-rendered keypoint channels to jointly model structure in the input images and target outputs.
- Learning Structured Outputs: The structured-output ablation evaluated left-knee prediction using channels for the knee alone, the knee and hip, or all keypoints.
5. Related Work
The paper situates IEF among structured-output learning, feedback-based vision, stacked inference, spatial-prior methods, and classical alignment models. It distinguishes IEF by iteratively correcting current estimates while learning representations over joint input-output spaces.
- Structured-output learning is established, but its combination with feature learning in computer vision is relatively modern.
- Feedback-based vision work has primarily addressed selective attention, whereas IEF uses feedback to model structured outputs.
- Table 4 reports MPII validation PCKh-0.5 for left-knee localization across IEF variants using different joint subsets and a direct-prediction baseline.
- Stacked inference methods differ from IEF because they may isolate outputs, use different models at each stage, or repeatedly predict answers from scratch.
- Multi-scale ConvNet methods capture long-range dependencies and spatial priors relevant to pose inference, including left-right body-side distinctions.
- Classical snakes and Active Appearance Models pursue related alignment goals but rely on hand-designed features or energy functions and can require many iterations.
6. Conclusions
IEF extends hierarchical representation learning from image inputs to structured output spaces by adding feedback that corrects current expectations. The paper identifies richer feedback rendering and deeper feedback injection as directions for higher-dimensional outputs such as 3D pose and segmentation.
- IEF addresses the asymmetry between hierarchical image representations and typically flat or hand-designed output representations.
- IEF shifts prediction toward correcting current expectations through a feedback connection in standard models.
- The pose example feeds back pose information only into the first ConvNet layer for simplicity, leaving possible benefits in mid-level layers unexplored.
- The experiments use Gaussian feedback renderings, while learned parametrized renderings are proposed for higher-dimensional outputs such as 3D pose or segmentation.