PaperlayerView exact revision on arXiv

Source-linked AI summary

Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition

Tianlang Chen, Chen Fang, Xiaohui Shen, Yiheng Zhu, Zhili Chen, Jiebo Luo

arXiv:2002.10322v5cs.CV

TL;DR

Video-based 3D pose estimation must resolve depth ambiguity from 2D keypoints while making effective use of temporal information. The paper decomposes pose into bone lengths and directions, combines global and local temporal modeling with joint-shift and visibility guidance, and reports state-of-the-art results on Human3.6M and MPI-INF-3DHP.

  • Problem

    3D pose estimation from 2D keypoints faces depth ambiguity, while temporal approaches may rely mainly on neighboring frames.

  • Method

    The model predicts bone lengths and directions separately, using global video frames for lengths, hierarchical fully-convolutional direction prediction, joint-shift coupling, and visibility scores.

  • Results

    The full model achieves state-of-the-art performance on the Human3.6M and MPI-INF-3DHP datasets.

  • Takeaways & Limitations

    Bone length and direction provide an effective intermediate representation bridging 2D keypoints and 3D joint locations.

Abstract

from arXiv · show

In this work, we propose a new solution to 3D human pose estimation in videos. Instead of directly regressing the 3D joint locations, we draw inspiration from the human skeleton anatomy and decompose the task into bone direction prediction and bone length prediction, from which the 3D joint locations can be completely derived. Our motivation is the fact that the bone lengths of a human skeleton remain consistent across time. This promotes us to develop effective techniques to utilize global information across all the frames in a video for high-accuracy bone length prediction. Moreover, for the bone direction prediction network, we propose a fully-convolutional propagating architecture with long skip connections. Essentially, it predicts the directions of different bones hierarchically without using any time-consuming memory units e.g. LSTM). A novel joint shift loss is further introduced to bridge the training of the bone length and bone direction prediction networks. Finally, we employ an implicit attention mechanism to feed the 2D keypoint visibility scores into the model as extra guidance, which significantly mitigates the depth ambiguity in many challenging poses. Our full model outperforms the previous best results on Human3.6M and MPI-INF-3DHP datasets, where comprehensive evaluation validates the effectiveness of our model.

I. INTRODUCTION

The paper addresses depth ambiguity and limited temporal context in video-based 3D pose estimation by decomposing pose into bone directions and lengths, enabling global video information for length prediction. It adds hierarchical convolutional direction prediction, joint-shift supervision, visibility guidance, and reports state-of-the-art performance on Human3.6M and MPI-INF-3DHP.

  • Motivation: The method targets depth ambiguity in lifting 2D keypoints to 3D and the limited use of neighboring frames by existing temporal approaches.Existing methods may use only a concentrated short segment of a video, even with large temporal windows.
  • Anatomy awareness: Decomposing 3D pose estimation into bone direction and bone length prediction exploits temporally consistent bone lengths and enables information from the entire video.The two sub-tasks are described as equivalent to predicting the original 3D joint locations.
  • Bone length prediction: The bone length branch uses a fully-connected residual network with network design and data augmentation mechanisms to prevent overfitting from few skeletons.The Human3.6M training set contains five actors corresponding to five bone-length settings.
  • Bone direction prediction: A fully-convolutional architecture predicts bone directions hierarchically, avoiding the high computation complexity of recurrent networks with long temporal windows.Less-variable torso bones guide predictions of more challenging limb bones.
  • Training and ambiguity reduction: Joint-shift loss coordinates the direction and length networks, while 2D keypoint visibility scores provide guidance for poses with occlusion or ambiguous relative depth.The visibility feature is intended to help when body parts are occluded or depth relations matter.
  • Results: The anatomy-inspired model achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP.The claim is reported as an overall contribution without numerical values in the supplied passages.

II. RELATED WORK

Related work includes direct image-to-3D approaches, 2D-keypoint lifting methods, and temporal models that improve consistency by exploiting information across video frames.

  • Image-based estimation: 3D human pose methods have included end-to-end CNNs that directly predict 3D joint locations from images.
  • 2D-to-3D lifting: A second category first detects 2D keypoints and then lifts them to 3D joint locations.
  • Temporal modeling: Video-based approaches exploit temporal information to reduce incoherent 3D pose predictions.Examples include temporal filtering and recurrent models that capture sequence-dependent context.

III. OUR MODEL

The anatomy-aware framework represents each pose through bone lengths and directions, using separate global and local temporal predictors whose outputs derive the 3D joints. A joint-shift loss couples the predictions, while the direction network uses hierarchical fully-convolutional propagation with long skip connections.

  • A. Anatomy-aware Framework: A human pose with j joints corresponds to j − 1 directed bones, and each joint location is derived from the lengths and directions along its path from the pelvis.The directed bones and joint representation are illustrated in Fig. 2.
  • A. Anatomy-aware Framework: The framework decomposes 3D joint prediction into bone length and bone direction prediction using separate sub-networks.Bone lengths use global video input for temporal consistency, whereas directions use a local temporal window.
  • A. Anatomy-aware Framework: The joint-shift loss compares ground-truth and predicted relative shifts for nonadjacent joint pairs, coupling the length and direction predictions.It provides joint supervision beyond the individual sub-network losses.
  • B. Bone direction prediction: The bone direction network uses long skip connections between adjacent sub-networks for hierarchical prediction.The architecture diagram specifies convolution stride, keypoint-set size, and input frame dimensions.

B. Bone Direction Prediction Network

The bone direction network uses temporal context and hierarchically predicts bone directions through stacked fully-convolutional sub-networks with long skip connections.

  • The network predicts current-frame bone directions from 2D keypoints across d consecutive frames, using adjacent-frame context.The current frame is placed at the center of the temporal input.
  • Bone directions are predicted hierarchically so simpler torso-bone predictions can guide more challenging limb-bone predictions.This follows the human skeleton anatomy and favors high-confidence predictions before subsequent ones.
  • The architecture stacks temporal fully-convolutional residual sub-networks and passes intermediate outputs to the next sub-network.
  • Long skip connections concatenate corresponding residual-block outputs between adjacent sub-networks, while each sub-network is trained independently.Backpropagation is blocked between different sub-networks.

C. Bone Length Prediction Network

The bone length network samples frames globally, predicts frame-level 3D joints, derives bone lengths, and uses self-attention and augmentation to reduce overfitting and unreliable-frame effects.

  • Randomly sampling l frames across the entire video captures global context without the prohibitive cost of processing every frame.The sampled frames feed a fully-connected residual network adapted from the direction-network structure.
  • The network predicts 3D joint locations for sampled frames under MPJPE supervision instead of directly predicting bone lengths, reducing overfitting from limited actor diversity.A randomly selected sampled frame supplies the MPJPE loss to balance convergence with the direction network.
  • Bone lengths are derived from each sampled frame’s predicted joints and reweighted with bone-specific self-attention because occlusion or foreshortening can make some frames unreliable.
  • Training augmentation creates variant bone lengths, modifies corresponding 3D joints, and reconstructs 2D keypoints using available camera parameters.

D. Overall Loss Function

The overall objective combines bone-direction, bone-length, joint-location, and joint-shift losses, with separate weights controlling their contributions.

  • The framework’s overall loss combines LD, LL, LJ, and LJS through weighted terms.The weights λD, λL, λJ, and λJS regulate the importance of each loss.
  • The joint shift loss supervises bone directions to match predicted bone lengths for long-range objectives, and updates only the bone direction network.The authors state that the anatomy-aware framework better exploits this loss than.
  • During inference, bone lengths for a target frame are predicted from randomly sampled video frames rather than all frames.Using all video frames does not improve performance according to the authors.

E. Incorporating the Visibility Scores

The model incorporates 2D keypoint visibility scores to address depth ambiguity, using pose-dependent implicit attention to adjust their influence.

  • Visibility scores provide extra guidance for ambiguous relative-depth cases such as crossed legs or hands near the chest or behind the back.
  • The model predicts each keypoint’s visibility score and incorporates it without feeding original images into the 3D estimator.The score represents confidence that the keypoint is visible in the frame.
  • Visibility importance depends on pose, so an implicit attention mechanism adaptively adjusts each score’s contribution.For example, hand visibility scores may be less useful when the hands are stretched far from the body.
  • A 1D convolution maps the visibility sequence to a temporal feature, which is multiplied element-wise with the 2D-keypoint feature before concatenation.The 2D-keypoint hidden feature can act as implicit attention weights for visibility importance.

IV. EXPERIMENTS

The experiments evaluate the model on Human3.6M and MPI-INF-3DHP using standard 3D pose metrics, with dataset-specific 2D keypoint and visibility inputs.

  • Datasets: Evaluation uses Human3.6M and MPI-INF-3DHP, covering constrained indoor, complex outdoor, and cross-dataset settings.Human3.6M uses five training subjects and two evaluation subjects, while MPI-INF-3DHP evaluates on an independent test set with different scenes, views, and actions.
  • Evaluation metrics: Human3.6M evaluation reports MPJPE, P-MPJPE, and MPJVE under Protocols 1 and 2 to measure accuracy and temporal smoothness.Protocol 1 uses unaligned mean per-joint position error, Protocol 2 uses similarity-aligned error, and MPJVE measures joint velocity error.
  • Evaluation metrics: MPI-INF-3DHP evaluation reports PCK within 150mm, AUC, and MPJPE on its independent test set.The test set differs from training in scenes, camera views, and relatively different actions, implicitly covering cross-dataset evaluation.
  • Implementation: The model uses predicted 2D keypoints from CPN on Human3.6M and AlphaPose on MPI-INF-3DHP, with AlphaPose visibility scores also provided.These inputs define the practical 2D-to-3D evaluation setting for both datasets.

C. Experiment results

Across Human3.6M and MPI-INF-3DHP, the full anatomy-aware model outperforms baselines, improves difficult poses and ground-truth-2D performance, and maintains practical efficiency.

  • Human3.6M results: On Human3.6M, the model achieves lower average Protocol 1, Protocol 2, and MPJVE errors, including gains of 3.4mm on Sitting and 5.6mm on Sitting down versus baseline.The model also produces smoother predictions and performs best on many actions.
  • Ground-truth 2D input: With ground-truth 2D keypoints, the model leaves a nearly 5mm Protocol 1 gap over the baseline, indicating further gains are possible with improved 2D detection.This experiment omits data augmentation because it would use extra ground-truth 2D keypoints.
  • Efficiency: A 9-frame model has similar computational complexity to the 243-frame baseline, while the full 2D-plus-3D pipeline is not bottlenecked by the proposed 3D model.Inference efficiency is compared using parameters, FLOPs, and FPS, with 2D keypoint detectors remaining the relevant speed reference.
  • Sensitivity analysis: Performance is insensitive to the tested hyperparameters, and two bone-direction sub-networks provide a trade-off between accuracy and computational complexity.Increasing the sub-network count continues to reduce MPJPE, but the final setting is two.
  • MPI-INF-3DHP results: The full model outperforms baselines by a large margin on MPI-INF-3DHP, although 81-frame models perform better than 243-frame models because the dataset has fewer training samples.The comparison is reported in Table V.
  • Model contribution: The proposed 3D model contributes beyond the 2D detector, as comparisons across Tables I, II, and V show superior performance is not only due to the detector.The comparison is made against baseline and spans estimated and ground-truth 2D inputs.

D. Ablation Study

The ablations show that explicit anatomy-aware decomposition, distant-frame bone-length prediction, long skip connections, joint-shift supervision, and implicit visibility attention each contribute to performance.

  • Module ablation: 1.4mm lower MPJPE reaches 46.3mm after MPJPE loss and data augmentation address bone-length overfitting, improving on the 47.7mm baseline.The joint shift loss, fully-convolutional propagating architecture, and attentive visibility feeding reduce error by about 0.5mm, 0.7mm, and 0.5mm, respectively.
  • Representation comparison: 45.8mm versus 47.4mm MPJPE favors explicit bone-length and bone-direction decomposition over generic bone-based representation.The comparison attributes the advantage to using global information across all frames and making better use of relative joint-shift supervision.
  • Frame selection: 46.7mm versus 45.8mm MPJPE shows that randomly sampled distant frames improve bone-length prediction over consecutive local frames.The experiment uses 50 consecutive local frames for the alternative model during training and inference.
  • Bone direction prediction: 45.8mm MPJPE after doubling temporal-network depth shows that long skip connections, rather than simply adding layers or parameters, are useful.The Baseline-D comparison removes long skip connections while retaining the deeper structure.
  • Visibility guidance: 44.8mm MPJPE for direct visibility-score concatenation supports implicit attention as a more effective way to feed visibility features.The comparison evaluates direct concatenation against the proposed attentive feeding mechanism.

E. Real-time 3D Pose Estimation

The real-time study compares causal baseline and three frame-selection strategies for bone-length prediction, balancing accuracy, recomputation, and inference speed.

  • Baselines: 49.8mm MPJPE characterizes BS-causal, which uses 81 consecutive frames with the current frame at the rightmost position.This causal baseline provides the real-time comparison point for the proposed strategies.
  • Frame selection: Ours-firstframe selects the first M video frames for bone-length prediction, avoiding iterative recomputation from frame M+1 onward.If M exceeds the current time index, it uses all frames before the current frame.
  • Frame selection: Ours-consecutive uses M consecutive local frames ending at the current frame, while all three proposed models are trained with randomly sampled 50-frame inputs.Training with inference-matched consecutive inputs does not further improve Ours-firstframe or Ours-consecutive.
  • Inference strategies: Ours-random reaches its best performance with a small M, and randomly sampling 50 frames is sufficient, but bone lengths must be predicted frame by frame.Ours-firstframe avoids repeated bone-length computation and is more efficient than Ours-random.
  • Speed trade-off: Ours-firstframe is about two times slower than BS-causal but more efficient than Ours-random because it avoids updating bone lengths frame by frame.Its bone-direction network is nearly twice as deep as the baseline.

V. CONCLUSION

The paper introduces anatomy-aware 3D pose estimation by decomposing joint prediction into bone lengths and directions, using global temporal information and hierarchical convolutional prediction.

  • Conclusion: The method predicts bone lengths and directions instead of directly regressing 3D joint locations.The resulting representation bridges 2D keypoints and 3D joint locations.
  • Conclusion: Bone-length prediction uses frames across the entire video with a fully connected residual network and bone-length re-weighting.This design exploits temporal information beyond local neighboring frames.
  • Conclusion: Bone-direction prediction uses long skip connections in a fully convolutional architecture for hierarchical prediction.The approach focuses on a human-pose learning representation grounded in skeleton anatomy.
  • Conclusion: Extensive experiments demonstrate that combining bone length and bone direction is an effective intermediate representation for 2D-to-3D pose estimation.The conclusion positions representation design as a fundamental aspect of human pose estimation.
Loading 2002.10322v5…