Source-linked AI summary
Chained Predictions Using Convolutional Neural Networks
Georgia Gkioxari, Alexander Toshev, Navdeep Jaitly
TL;DR
The paper addresses structured vision prediction without hand-designing conditional dependencies among output variables. It introduces chained neural predictions for pose estimation, with CNN-based spatial prediction and alternative tied or untied weights, achieving top-performing results on image and video pose benchmarks.
Problem
Existing structured prediction models require hand-designed dependencies and conditional-independence assumptions to make inference tractable.
Method
The paper sequentially predicts body-part locations, conditioning each prediction on previously predicted parts, using end-to-end CNNs with multi-scale deconvolution and tied or untied predictor weights.
Results
86.1% PCKh on MPII and 91.8% PCK on PennAction, with top-performing results for image and video pose estimation.
Takeaways & Limitations
Chained predictions can model structured vision outputs without assumptions about the joint distribution beyond neural-network capacity for conditional distributions.
Takeaways & Limitations
The pose experiments use a fixed joint ordering motivated by human-body kinematics, rather than learning the ordering for the task.
Abstract
from arXiv · showhide
In this paper, we present an adaptation of the sequence-to-sequence model for structured output prediction in vision tasks. In this model the output variables for a given input are predicted sequentially using neural networks. The prediction for each output variable depends not only on the input but also on the previously predicted output variables. The model is applied to spatial localization tasks and uses convolutional neural networks (CNNs) for processing input images and a multi-scale deconvolutional architecture for making spatial predictions at each time step. We explore the impact of weight sharing with a recurrent connection matrix between consecutive predictions, and compare it to a formulation where these weights are not tied. Untied weights are particularly suited for problems with a fixed sized structure, where different classes of output are predicted in different steps. We show that chained predictions achieve top performing results on human pose estimation from single images and videos.
1 Introduction
The paper adapts sequence-to-sequence prediction to structured vision outputs, using chained neural predictions so each output depends on earlier ones. It applies this approach to human pose estimation with CNN-based spatial predictions and evaluates shared and untied predictor weights.
- Motivation: Structured prediction captures dependencies among output variables, but traditional models require hand-designed structures and conditional-independence assumptions for tractable inference.Human pose estimation is an example where one body part’s location is constrained by many others.
- Motivation: Sequence-to-sequence models predict outputs in order, conditioning each prediction on the input and all previous outputs through a recurrent network.This decomposes the output-sequence probability with the chain rule and permits parameter sharing across steps.
- Method: The proposed chain model predicts body-part locations sequentially, with each prediction depending on previously predicted parts and learned end-to-end CNN features.CNNs process inputs and multi-scale deconvolutional networks produce spatial outputs.
- Method: The paper compares untied predictors for semantic-specific information flow with shared weights that enforce recurrence across video predictions.The shared-weight formulation is an RNN similar to the sequence-to-sequence model.
- Results: 86.1% PCKh is achieved on MPII, while 91.8% PCK is reported as state of the art on PennAction.These results cover single-image and video human pose estimation.
2 Related Work
Related work frames chained prediction as sequential structured-output modeling, contrasting it with graphical models and iterative refinement. The paper uses ordering to factorize predictions without imposing conditional-independence structure on the joint distribution.
- Sequential structured prediction: SEARN and sequence-to-sequence models generate outputs sequentially by conditioning each next prediction on prior outputs and the input.Sequence-to-sequence models use recurrent networks and next-step losses across output positions.
- Output ordering: For pose estimation, the paper uses a fixed head-to-torso-to-extremities ordering motivated by human-body kinematics.The authors note that intuitive orderings can slightly improve some problems, while simpler problems may tolerate many orderings.
- Graphical models: Traditional pose models often assume each body part is conditionally independent of all others except its kinematic parent.This simplification can reduce performance and motivates richer dependency modeling.
- Chain model: The chain model uses ordering only to decompose the joint probability and does not assume a simplified dependency structure among output variables.Correlations are learned through nonlinear neural operations rather than hand-designed binary feature terms.
- Iterative refinement: Earlier refinement methods first make independent predictions for all parts, whereas chained prediction makes later outputs depend on earlier predictions from the start.The paper notes that combining chaining with iterative refinement could improve results.
3 Chain Models for Structured Tasks
Chain models adapt sequential prediction to structured vision tasks by conditioning each output on the input and earlier outputs. CNN-based hidden states and task-specific weight sharing support both single-image and video predictions.
- Each output prediction is conditioned on the input and all previously predicted outputs, allowing the chain to represent structured dependencies incrementally.The hidden state carries input features, prior states, and previous output values before producing the next conditional probability.
- 3.1 Chain Models for Single Images: For single images, the model predicts object or body-part locations using an input-derived hidden state and convolutional neural networks for spatial outputs.The initial hidden state is computed as h0 = CNN(X), while location probabilities are produced from the hidden state.
- 3.1 Chain Models for Single Images: Previous output variables are transformed by a neural network and combined with the hidden state before a nonlinear update and location-scoring softmax.In image applications, the hidden state is a feature map, convolution replaces multiplication, and e is implemented as a CNN.
- 3.2 Chain Models for Videos: Parameters may remain untied across steps for fixed-size tasks whose outputs differ, whereas video applications tie predictors because they repeatedly estimate the same information.For videos, tied parameters produce a convolutional recurrent neural network.
- 3.2 Chain Models for Videos: In videos, predictions for the current frame use features from its image together with past images and past output variables.The hidden-state update adds CNN features extracted from the current image, and a larger time horizon can ease learning.
- Training can replace some ground-truth previous outputs with samples from the model distribution, although inaccurate early predictions make this substitution counterproductive at initialization.The substitution is described as a way to mitigate overfitting in sequence-to-sequence models.
4 Experimental Evaluation
The experiments evaluate chained pose prediction on single images and videos, using CNN encoders, multi-scale deconvolutional decoders, and beam-search inference. Chaining improves over comparable baselines, especially for difficult joints and temporal pose localization.
- Datasets: MPII single-image experiments use about 40K annotated people, while Penn Action video experiments use 2326 sports sequences.MPII provides up to 16 joints per frame; Penn Action provides up to 13.
- Network architecture: CNNy converts each hidden state into a joint-location heatmap using two multi-scale deconvolutional towers and a softmax probability distribution.The towers increase feature-map dimensions by a factor of 2 and combine different filter sizes.
- Single-image pose estimation: The model predicts joints sequentially, with a fixed order that processes easier joints first and harder joints such as Wrist and Ankle later.The ordering is motivated by joint detection rates, allowing later predictions to use contextual information from earlier joints.
- Inference: Beam search approximates inference over (HW)^T possible joint configurations by retaining the best B partial sequences at each step.Full-pose quality is measured by the sum of individual joint log-probabilities.
- Single-image results: The chain model significantly improves over feed-forward baselines on MPII, with the largest gains for Wrists and Ankles.Error analysis attributes reductions for these joints to fewer confusions with other joints and the background.
- Video results: On Penn Action, chaining improves localization across all joints over per-frame CNN and RNN baselines, with larger gains for arms and best performance at TH = 3.The chain model also disambiguates right-left confusions that frequently affect per-frame detection.
5 Conclusions
Chained predictions provide a powerful approach for structured vision tasks by avoiding assumptions about the output variables’ joint distribution. Experiments show top-performing pose-estimation results on images and videos.
- Chained predictions avoid assumptions about the joint distribution of output variables, apart from neural-network capacity to model conditional distributions.
- The approach is presented as a powerful tool for structured vision tasks.
- Experiments demonstrate top-performing results for pose estimation from images and videos.