PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Feature Learning for Pose Estimation

Xiao Chu, Wanli Ouyang, Hongsheng Li, Xiaogang Wang

arXiv:1603.09065v1cs.CV

TL;DR

Human pose estimation must localize joints despite varied appearance, viewpoint, occlusion, and clutter. The paper addresses this by jointly learning correlated joint features with geometrical transform kernels and bi-directional tree message passing, achieving strong PCP results with simple post-processing.

  • Problem

    Score maps and predicted labels omit rich joint information, limiting structural learning for pose estimation.

  • Method

    The framework jointly learns feature-map relationships with geometrical transform kernels and passes information between correlated joints through a bi-directional tree.

  • Results

    18% mean PCP improvement on FLIC over separately learned ConvNet features accompanies best mean PCP results of 80.8% on LSP and 95.2% on FLIC.

  • Takeaways & Limitations

    Feature-level structural learning substantially improves feature learning and outperforms the state of the art with very simple post-processing.

Abstract

from arXiv · show

In this paper, we propose a structured feature learning framework to reason the correlations among body joints at the feature level in human pose estimation. Different from existing approaches of modelling structures on score maps or predicted labels, feature maps preserve substantially richer descriptions of body joints. The relationships between feature maps of joints are captured with the introduced geometrical transform kernels, which can be easily implemented with a convolution layer. Features and their relationships are jointly learned in an end-to-end learning system. A bi-directional tree structured model is proposed, so that the feature channels at a body joint can well receive information from other joints. The proposed framework improves feature learning substantially. With very simple post processing, it reaches the best mean PCP on the LSP and FLIC datasets. Compared with the baseline of learning features at each joint separately with ConvNet, the mean PCP has been improved by 18% on FLIC. The code is released to the public.

1. Introduction

The paper proposes learning correlations among body-joint feature maps rather than relying only on score maps, whose compressed information misses pose and appearance details. An end-to-end geometrical-kernel and bi-directional tree framework improves pose-estimation results.

  • Human pose estimation remains difficult because limb orientation, clothing, viewpoint, clutter, truncation, and occlusion vary substantially.
  • Score maps summarize each location as a probability and omit detailed cues such as joint rotation, limb orientation, clothing, and occlusion.
  • Different visual patterns activate different feature channels, while feature maps across joints exhibit spatial and semantic correlations.
  • Geometrical transform kernels capture spatial and co-occurrence relationships among feature maps and can be implemented with convolution in end-to-end learning.
  • A bi-directional tree-structured model passes messages along correlated joints in both directions, allowing each joint to receive information from neighboring joints.
  • 18% mean PCP improvement on FLIC is achieved over separately learned ConvNet features, while mean PCP reaches 80.8% on LSP and 95.2% on FLIC.

2. Related Works

Prior pose-estimation work largely separates structural modeling from feature generation. One group models joint geometry on score maps or labels, while another uses increasingly powerful ConvNet-based feature generators and joint prediction systems.

  • One group models the geometrical distribution of body joints, often as post-processing on detection score maps and prediction labels using hand-crafted features.
  • Pictorial Structure Models use pairwise terms for joint locations, while flexible mixture-of-parts models combine part detections with tree-structured inference.
  • A second group develops stronger feature generators such as ConvNets for regression, pairwise relations, local-global prediction, and multi-resolution joint training.

3.1. Feature maps of body joints

The model uses a fully convolutional VGG backbone to produce joint-specific feature maps. These maps retain visual and pose information across images, with separate channels responding to recurring joint-related patterns.

  • ConvNets learn hierarchical representations, and this model extracts feature maps from the fcn7 layer of a fully convolutional VGG network.
  • Each body joint has a separate set of 128 feature maps, while all joints share lower layers through the fcn6 layer.
  • The shared fcn6 representation is a 4,096-dimensional feature vector at each spatial location.
  • The fcn7 feature vector for each joint is computed using joint-specific filters and bias from the shared fcn6 features.
  • The same feature channels respond differently across images, with high responses aligning to recurring neck, wrist, or lower-arm patterns.

3.2. Information passing

The framework passes information between joint feature maps using jointly learned geometrical transform kernels and a bi-directional tree. Kernels spatially align correlated responses before updating features, while opposite tree directions provide complementary information.

  • Kernel-based information passing: Geometrical transform kernels pass messages between feature maps through convolution, with the FCN filters and kernels learned jointly.The kernels can shift feature responses spatially before information from one joint is incorporated into another.
  • Kernel-based information passing: A correlated lower-arm feature map is shifted toward the elbow and added to an elbow feature map, improving the refined prediction.Direct addition is avoided because the two joints have a spatial mismatch; negative kernel values can suppress anti-correlated channels.
  • Stacked transform kernels: Successive convolutions approximate large transform kernels, reducing parameters while supporting information transfer between distant neighboring joints.On FLIC, a target within 72 pixels can be reached using three successive 7 × 7 kernels.
  • Bi-directional tree: The tree propagates information through nearby, relatively stable joint relationships instead of directly connecting every pair of joints.A fully connected graph would require difficult-to-learn large kernels and would include unstable relationships such as head-to-foot connections.
  • Bi-directional tree: In the upward branch, information flows from leaf joints to root joints, updating each joint from refined neighboring features where the tree provides incoming messages.Leaf features that receive no messages remain unchanged, while intermediate joints aggregate transformed information from their children.
  • Bi-directional tree: A second branch reverses the tree flow, and the two updated feature sets are concatenated to form the final feature representation at each joint.The two branches provide complementary features from opposite information-flow directions.

3.3. Model analysis

The model combines information across receptive-field sizes through message passing and uses a bi-directional tree within the pose-estimation pipeline. Its transform kernels are substantially more expressive than score-map message passing, while joint-specific feature semantics motivate unshared weights.

  • Enlarged receptive field: Message passing combines features at multiple scales because transformed features have larger receptive fields and are added to neighboring joints’ original features.Feature maps in VGG fcn7 have 188 × 188 receptive fields, while transformed features reach 332 × 332 before combination.
  • Pipeline: The pose-estimation pipeline refines original joint feature maps with a bi-directional tree, then predicts joint score maps from the refined features.Figure 4 distinguishes copy operations from convolution in the pipeline.
  • Network configuration: The network table specifies joint-specific filter banks and 17 × 2 bi-directional-tree connections for message passing.The table denotes element-wise summation and directs the message-passing process to the two opposite tree flows.
  • Expressive power: The transform kernels have greater expressive power than existing score-map message passing; on LSP, each joint pair uses 128 × 64 × 2 kernels of size 7 × 7.The message-passing process also increases model depth, with the root joint having 34 layers and multiple intermediate supervision.
  • Weight sharing: Unlike an RNN, the model does not share feature channels or geometrical transform kernels across joints because joint channels have different semantic meanings and part-specific relationships.RNNs share feature channels across time steps, whereas this model assigns joints their own channels and unshared transforms.

4. Summary of Pipeline

The pipeline uses a VGG-16-based fully convolutional model with joint-specific feature channels, bidirectional tree message passing, and simple score-map post-processing. It produces joint score maps while handling multi-person ambiguities through spatial constraints.

  • Network pipeline: VGG-16 removes pool4 and pool5 to preserve resolution, producing 56 × 56 joint score maps from 448 × 448 inputs.Channel dropout after ReLU6 is used to prevent overfitting.
  • Structured feature learning: Each joint receives 128 feature channels per message-passing direction, with all joints sharing layers through fcn6.The structured feature learning layer refines these maps using a bi-directional tree and concatenates complementary directional features.
  • Post-processing: A distance descriptor constrains joint separations during post-processing because independently selecting score-map maxima can link joints from different people.The descriptor uses fixed weights [0.01, 0.01].
  • Evaluation: PDJ curves compare elbows and wrists on FLIC across MODECT, Deep pose, Tompson et al., Chen&Yuille, and the proposed method.The curves are presented as a cross-data comparison.

5. Training

Training formulates joint localization as pixel-level classification over joint and background classes. A mask reduces the influence of abundant negative samples during optimization.

  • Training objective: Joint localization is formulated as a classification problem with a 56 × 56 × 19 supervision tensor.The tensor represents labels at each pixel location.
  • Training objective: The 19 classes comprise 18 human body joints and one background class.Each pixel receives a class label.
  • Negative-sample handling: The binary mask m randomly retains only 0.05% of negative samples because negative training samples far outnumber positive ones.The mask is applied during the objective calculation.

6. Experimental Results

Experiments on FLIC and LSP evaluate strict PCP, PDJ, model components, and qualitative predictions. The proposed method outperforms prior approaches and shows gains from structured feature learning and bidirectional message passing.

  • Overall results on FLIC: 95.2% mean PCP on FLIC is 3.3 points higher than the previously best approach.Previous works only evaluate upper and lower arms, so the reported mean averages those two parts.
  • Overall results on FLIC: The proposed method outperforms all previous methods across every normalized precision threshold in FLIC PDJ curves for elbows and wrists.PDJ plots percentage of correctly detected joints against a ground-truth-pose-scale-normalized precision threshold.
  • Component investigation: 13% mean PCP improvement over the baseline results from single upward message passing on FLIC.The single-direction model achieves 87.9% mean PCP versus the baseline result of 75.1.
  • Component investigation: 93.4% PCP from the bidirectional tree model is a 5.4% improvement over the single-branch model.Performance for each body joint is consistently improved compared with the preceding experiments.
  • Additional analysis: Multiple score maps are additionally produced by clustering each body joint into appearance mixtures.The mixtures use relative joint-to-parent locations normalized by head scale.
  • Results on LSP: The method outperforms previous state-of-the-art by 5.8% on LSP and obtains the best result on every evaluated body part.Qualitative results are shown for FLIC and LSP, with FLIC in the first row and LSP in the second and third rows.

7. Conclusion

The paper concludes that modeling correlations among body-joint feature maps improves pose estimation. Geometrical transform kernels and bidirectional tree message passing support richer feature learning with simple post processing.

  • Conclusion: Feature-level information passing provides more detailed body-joint descriptions than score maps and is implemented with geometrical transform kernels.The kernels capture correlations among feature maps and can be used within convolutional layers.
  • Conclusion: A bidirectional tree structure learns complementary features by passing information between joints in opposite directions.The framework jointly models feature learning and relationships between body joints.
  • Conclusion: On two public datasets, the framework substantially improves feature learning and outperforms state-of-the-art methods with very simple post processing.The conclusion identifies more advanced post processing and alternative message-passing structures as directions for further improvement.
Loading 1603.09065v1…