Source-linked AI summary
Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
Xianjie Chen, Alan Yuille
TL;DR
Articulated pose estimation from a single image requires modeling both body-part appearance and variable spatial relations. The paper introduces a graphical model with image dependent pairwise relations, learned by DCNNs, and reports significant gains on LSP and FLIC plus strong Buffy generalization without Buffy training.
Problem
Articulated human pose estimation from a single static image must represent local part evidence and spatial relationships among body parts.
Method
A graphical model uses local image patches for both unary part detection and mixture-based pairwise relations, with DCNNs learning conditional probabilities and S-SVM learning score weights.
Results
The method significantly outperforms state-of-the-art methods on LSP and FLIC and performs very well on Buffy without training on that dataset.
Takeaways & Limitations
Image-dependent pairwise relations combine graphical-model structure with DCNN efficiency and support strong cross-dataset generalization.
Abstract
from arXiv · showhide
We present a method for estimating articulated human pose from a single static image based on a graphical model with novel pairwise relations that make adaptive use of local image measurements. More precisely, we specify a graphical model for human pose which exploits the fact the local image measurements can be used both to detect parts (or joints) and also to predict the spatial relationships between them (Image Dependent Pairwise Relations). These spatial relationships are represented by a mixture model. We use Deep Convolutional Neural Networks (DCNNs) to learn conditional probabilities for the presence of parts and their spatial relationships within image patches. Hence our model combines the representational flexibility of graphical models with the efficiency and statistical power of DCNNs. Our method significantly outperforms the state of the art methods on the LSP and FLIC datasets and also performs very well on the Buffy dataset without any training.
1 Introduction
The paper introduces image dependent pairwise relations (IDPRs), using local image patches to detect body parts and predict their spatial relationships. A DCNN supplies these image-dependent terms within a graphical model, improving pose estimation across standard benchmarks and cross-dataset evaluation.
- Prior approach: The graphical-model formulation builds on unary appearance terms and pairwise contextual relations between body parts.This is the dominant modeling framework described for pose estimation in prior work.
- Motivation and contribution: IDPRs use local image patches to predict both part presence and the relative positions of neighboring parts.This makes pairwise terms adaptive rather than fixed across images.
- Motivation and contribution: The model combines graphical models’ spatial-relation flexibility with DCNNs’ data-driven representation and feature-sharing capabilities.The DCNN estimates part presence and spatial relationships for unary and pairwise terms.
- Experimental outcome: The method significantly outperforms state-of-the-art methods on the LSP and FLIC datasets and obtains strong cross-dataset results on Buffy without training there.The Buffy evaluation tests generalization from training on other data.
2 The Model
The model represents pose with a tree-structured graphical model whose unary and pairwise scores depend on local image patches. Pairwise relations are mixtures of spatial types, while a DCNN jointly models image-dependent part and relation information.
- Graphical model variables: A tree-structured graph represents part locations at nodes and spatial relationships between related parts at edges.Each part location is a pixel coordinate, and each edge carries discrete relation types.
- Unary terms: Unary terms provide local evidence that part i occurs at location l_i using the image patch centered at that location.The appearance function is parameterized by θ and weighted by w_i.
- Image-dependent pairwise terms: IDPR terms discretize neighboring parts’ relative positions into multiple relation types with learned mean positions and quadratic deformation features.The relation score uses image-dependent terms together with deformation and weight parameters.
- Full score: The full score is the sum of unary and pairwise terms over part locations, relation types, and the input image.The model includes a scalar bias parameter w_0.
- Image-dependent terms and DCNNs: A DCNN learns the conditional distribution of which part is present and the relation types associated with its neighbors from a local image patch.The state space includes background and combines relation types according to each part’s number of neighbors.
- Relations to standard models: The formulation contains pictorial-structure and mixtures-of-parts models as restricted cases, while jointly modeling unary and image-dependent terms in one DCNN.The CRF comparison emphasizes the joint treatment of image-dependent terms.
3 Inference
Inference searches for the part locations and relation types maximizing the model score. Because the relational graph is a tree, dynamic programming solves this optimization efficiently, with distance transforms accelerating quadratic pairwise maximization.
- Optimization: The optimal pose configuration maximizes the score over all part locations and relation types.The output is the pair (l*, t*) that achieves the highest full score.
- Dynamic programming: Dynamic programming recursively computes maximum subtree scores and recovers the optimal configuration through a backward pass.Each subtree score considers child locations, relation types, pairwise scores, and child-subtree scores.
- Computational efficiency: Generalized distance transforms accelerate maximization over child locations because the pairwise term is quadratic.Under equal relation-type counts, the resulting complexity is O(T^2LK).
- Computational efficiency: A single sliding DCNN computes image-dependent terms over all locations while sharing computations across overlapping regions.This makes image-dependent term computation efficient.
4 Learning
Learning separates relation-type geometry, image-dependent predictions, and model weights. Relative-position clusters define relation types, a DCNN learns conditional probabilities from labeled patches, and an S-SVM learns the score weights.
- Parameter learning: The model learns mean relative positions, image-dependent parameters, and score weights separately using K-means, a DCNN, and an S-SVM.These correspond respectively to r, θ, and w.
- Relation-type learning: K-means clusters annotated relative positions into relation types, with each cluster center serving as a mean relative position.The experiments use T_ij = 11 clusters for every pairwise relation.
- DCNN learning: Annotated local patches receive labels for the visible part and its relation types with neighboring parts.Background patches are sampled from negative images.
- DCNN learning: A multi-class DCNN trained with softmax loss outputs p(c, m_N(c)|I(l_i); θ) for the image-dependent terms.The network uses five convolutional layers, two max-pooling layers, and three fully connected layers.
- Weight learning: An S-SVM learns the weight parameters from positive and negative structured examples using a 0-1 loss.The feature vector concatenates learned image-dependent terms, deformation features, and a constant.
5 Experiment
Experiments evaluate the method on LSP and FLIC using strict PCP, plus PDJ on FLIC, and test cross-dataset generalization on Buffy. The method significantly outperforms prior approaches on the benchmarks, generalizes strongly to Buffy without Buffy training, and benefits from image-dependent pairwise terms.
- Datasets and Evaluation Metrics: Experiments use LSP full-body poses and FLIC upper-body poses, with negative training images from INRIAPerson.LSP has 1000 training and 1000 testing images; FLIC has 3987 training and 1016 testing images.
- Datasets and Evaluation Metrics: Strict PCP requires both segment endpoints to lie within 50% of the ground-truth segment length, while FLIC evaluation also reports scale-invariant PDJ curves.PDJ varies the localization threshold and normalizes it by the distance between the left shoulder and right hip.
- Experimental Setup: The DCNNs use 36 × 36-pixel LSP patches and 54 × 54-pixel FLIC patches, with five convolutional, two max-pooling, and three fully connected layers.The final output has |S| dimensions; dropout, local response normalization, and overlapping pooling are also used.
- Benchmark Results: The method significantly outperforms state-of-the-art methods on strict PCP for both LSP and FLIC, with a 5.8% average advantage over the best previous LSP result.The LSP comparison notes that DeepPose uses Person-Centric annotations and 10,000 extra training images.
- Cross-dataset Generalization: On Buffy, applying the FLIC-trained model without Buffy training significantly outperforms prior methods under both PCP criteria and shows strong cross-dataset generalization.The margin over DeepPose increases on Buffy relative to FLIC, implying better generalization to Buffy.
- Terms Analysis: Diagnostic experiments show that unary-only and image-independent pairwise terms are weaker, while adding IDPR terms raises final strict PCP performance to 75.0% on LSP.The No-IDPRs variant uses a mixture of Gaussian deformations with image-independent biases.
6 Conclusion
The paper uses local image measurements to detect body parts and predict their spatial relationships through Image Dependent Pairwise Relations, combining graphical models with DCNNs. It reports strong performance on LSP and FLIC, and on Buffy without training.
- Image Dependent Pairwise Relations use local image patches both to detect parts and to predict spatial relationships represented by a mixture model.
- The model combines graphical models’ spatial-relation representation with DCNNs’ efficiency and statistical power.
- The method outperforms state-of-the-art methods on the LSP and FLIC datasets.
- The method performs very well on the Buffy dataset without any training.
- Figure 5 shows part-localization results and the model’s graph skeleton, with failures typically caused by foreshortening, occlusions, clothing, or overlapping people.