PaperlayerView exact revision on arXiv

Source-linked AI summary

SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach

Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, Stephen Lin

arXiv:2007.09389v1cs.CV

TL;DR

Rare and unseen poses are difficult because they have few global training examples, despite being composed of recurring local configurations. SRNet processes local joint groups separately while recombining low-dimensional global context, and it improves state-of-the-art performance, especially on rare and unseen poses.

  • Problem

    Rare and unseen poses are difficult to predict because few training examples limit the network’s ability to model them, while synthetic augmentation may have a domain gap.

  • Method

    SRNet splits the body into local joint groups processed in separate branches and recombines low-dimensional global context, with an analogous channel-wise design for temporal convolutions.

  • Results

    SRNet outperforms state-of-the-art techniques, especially for rare and unseen poses, with competitive single-frame and superior video-input performance.

  • Takeaways & Limitations

    The split-and-recombine design improves generalization to global poses that are rare or absent from the training set.

  • Takeaways & Limitations

    The standard Human3.6M Subject Protocol does not examine rare poses well because subjects perform fixed actions in similar ways.

Abstract

from arXiv · show

Human poses that are rare or unseen in a training set are challenging for a network to predict. Similar to the long-tailed distribution problem in visual recognition, the small number of examples for such poses limits the ability of networks to model them. Interestingly, local pose distributions suffer less from the long-tail problem, i.e., local joint configurations within a rare pose may appear within other poses in the training set, making them less rare. We propose to take advantage of this fact for better generalization to rare and unseen poses. To be specific, our method splits the body into local regions and processes them in separate network branches, utilizing the property that a joint position depends mainly on the joints within its local body region. Global coherence is maintained by recombining the global context from the rest of the body into each branch as a low-dimensional vector. With the reduced dimensionality of less relevant body areas, the training set distribution within network branches more closely reflects the statistics of local poses instead of global body poses, without sacrificing information important for joint inference. The proposed split-and-recombine approach, called SRNet, can be easily adapted to both single-image and temporal models, and it leads to appreciable improvements in the prediction of rare and unseen poses.

1 Introduction

Rare and unseen poses are difficult because training data contains few global examples, while their local joint configurations may recur across other poses. SRNet exploits this structure by learning local regions with limited global context, improving generalization to rare and unseen poses.

  • Rare or unseen poses are hard to predict because few training examples limit the network’s ability to model them.
  • Synthetic augmentation can suffer from a real-versus-synthetic domain gap, while viewpoint simulation provides only a narrow range of pose variation.
  • Rare global poses can contain local joint configurations that appear frequently across other training poses, such as bicycling-like legs in stair climbing or marching.
  • SRNet splits the body into locally interdependent joint groups, processes each group separately, and recombines low-dimensional context from other groups.
  • Reducing non-local context limits its influence on local feature learning while retaining enough global information to avoid incoherent pose estimates.
  • SRNet is competitive for single-frame input, surpasses state-of-the-art techniques on video input, and improves performance on rare and unseen poses.

3 Method

SRNet decomposes global 3D pose estimation into local joint-group problems while restoring reduced global context to preserve coherence. The design applies to both fully connected and temporal convolutional models.

  • SRNet decomposes global pose estimation into several local pose estimation problems.
  • Body Partitioning into Local Pose Regions: The body is partitioned into regions whose joints are strongly interdependent internally and less dependent on joints outside each region.
  • Group connected layers remove inter-group connections so local features reflect only their region’s training distribution, but they can lose global coherence.
  • SRNet: Incorporating Low-Dimensional Global Context: Low-Dimensional Global Context coarsely represents less-relevant joints and recombines that information with local features while limiting disruption to local modeling.
  • SRNet: Incorporating Low-Dimensional Global Context: The mapping function gates non-local information between fully connected, group connected, and low-dimensional intermediate regimes.
  • FC, GP, LF, ES, and SFS modify where feature sharing is cut or restored, providing baselines for the split-and-recombine structure.
  • For temporal models, SRNet applies the same strategy across convolution channels, reducing other groups’ channels before concatenating them as context.

4 Datasets and Rare-Pose Evaluation Protocols

The evaluation uses Human3.6M and MPI-INF-3DHP with established pose-estimation metrics, while several protocols specifically test generalization to rare or unseen poses. The Rare Pose Protocol selects the least-occurring test poses using pose similarity and occurrence measures.

  • Datasets and metrics: Human3.6M contains 3.6 million video frames from four camera viewpoints and 15 activities, with 3D joint locations obtained from motion capture.
  • Datasets and metrics: MPI-INF-3DHP includes constrained indoor and complex outdoor scenes and covers more diverse poses and actions than Human3.6M.
  • Datasets and metrics: Human3.6M evaluation uses MPJPE and PA-MPJPE, while MPI-INF-3DHP uses PCK at 150mm and AUC across PCK thresholds.
  • Evaluation protocols: The standard Subject Protocol trains on subjects S1, S5, S6, S7, S8 and tests on S9, S11, but does not adequately examine rare poses because actions are performed similarly.
  • Evaluation protocols: Cross Action trains on one Human3.6M action and tests on all actions, while Cross Dataset applies Human3.6M-trained models to MPI-INF-3DHP.
  • Rare Pose Protocol: The Rare Pose Protocol trains on all poses from five subjects, tests on selected poses from S9 and S11, and evaluates the rarest poses.
  • Rare Pose Protocol: Pose similarity compares corresponding joints with a Gaussian falloff, and occurrence averages similarity to all poses before selecting the R% with lowest occurrence for evaluation.

5 Experiments

Experiments show that combining local feature learning with low-dimensional global context improves 3D pose estimation, especially for rare poses, across ablations and benchmark settings.

  • 5.1 Ablation Study: Local feature learning and local inference each help, but replacing all fully connected layers with group layers sharply degrades performance.The result indicates that local regions still require information from other body regions.
  • 5.1 Ablation Study: 7.4mm, 16.8mm, and 20.9mm improvements over FC occur on the Subject, Rare Pose 20%, and Rare Pose 10% protocols, respectively.The corresponding relative improvements are 15.8%, 22.1%, and 23.5%.
  • 5.1 Ablation Study: 2.8mm, 10.6mm, and 14.5mm improvements over SFS occur on the Subject, Rare Pose 20%, and Rare Pose 10% protocols, respectively.The relative improvements are 7.1%, 17.9%, and 21.2%; gains increase for rarer poses.
  • 5.1 Ablation Study: Removing global context produces the worst performance, while lower-dimensional context performs better and a single dimension achieves the best result.These findings support retaining global information in compressed form.
  • 5.1 Ablation Study: Performance initially improves as the number of local groups increases, but drops beyond five groups because groups may contain fewer than three joints.Randomly shuffling joints among groups also decreases performance, supporting the importance of physical joint relationships.
  • 5.2 Comparison with State-of-The-Art Methods: SRNet achieves the best cross-dataset performance on MPI-INF-3DHP, improving over the state of the art by 4.9% on PCK and 19.3% on AUC.The comparison trains on Human3.6M and tests on MPI-INF-3DHP.
  • 5.2 Comparison with State-of-The-Art Methods: The single-frame model improves over by 11.4mm, while the temporal model improves from 37.2mm to 32.0mm with 2D ground truth input.The single-frame model performs better on 93% of actions; the temporal result is a 14.0% relative improvement.

6 Conclusion

The paper concludes that SRNet improves generalization in 3D human pose estimation by splitting joints into local groups and recombining low-dimensional global context.

  • 6 Conclusion: SRNet outperforms state-of-the-art techniques, especially for rare and unseen poses.Its key design splits the body into local joint groups and recombines low-dimensional global context.

Estimation with a Split-and-Recombine Approach

The supplementary material provides implementation details, additional experiments, and qualitative results omitted from the main paper.

  • Estimation with a Split-and-Recombine Approach: The supplement covers experiment settings, combination operators for Equation 5, and comparisons using MPJPE and PA-MPJPE.It also includes additional experimental and qualitative results.

1 Implementation Details

Experiments use either 2D ground-truth keypoints or detections from a pretrained, Human3.6M-finetuned CPN-based detector, with a fixed residual fully connected temporal architecture and scheduled optimization.

  • 1 Implementation Details: The model uses 2D ground truth or off-the-shelf detector outputs, with no extra 2D data used for mixed training.The detector is a smoothed CPN model pretrained on COCO and finetuned on Human3.6M.
  • 1 Implementation Details: Training uses Amsgrad for 80 epochs, a 0.001 initial learning rate with 5% per-epoch decay, 1024 channels, normalization, Leaky ReLU, and eight stacked layers.Residual connections wrap every two intermediate layers.

2 Design Choices for the Combination Operator

The study compares addition, multiplication, and concatenation for recombining low-dimensional global contexts. Addition and multiplication outperform the FC and SFS baselines.

  • Equation 5 recombines low-dimensional global contexts into local groups using the combination operator ◦.The operator is implemented as multiplication by default.
  • Addition and multiplication obtain favorable results and surpass the FC and SFS baselines.The comparison evaluates alternative combination operators under the Subject protocol using MPJPE.
  • The operator comparison also reports the number of learnable parameters for each model.

3 More Results on Human3.6M

SRNet achieves state-of-the-art Human3.6M results across input settings and evaluation metrics, including cross-action generalization. Detailed comparisons report improvements for both detected and ground-truth 2D inputs.

  • Human3.6M comparisons: SRNet achieves the new state-of-the-art with either 2D keypoint detection or 2D ground-truth input.The comparisons cover single-image and temporal settings on Human3.6M.
  • MPJPE results: 33.9mm MPJPE improves on 's 36.3mm result with 2D ground-truth input, a relative 6.6% improvement.
  • PA-MPJPE results: 24.3mm PA-MPJPE improves on 's 27.9mm result with 2D ground-truth input, a relative 14.8% improvement.
  • PA-MPJPE results: With detected 2D keypoints, SRNet improves PA-MPJPE over from 36.5mm to 34.9mm, a relative 4.4% improvement.
  • Cross-action results: Cross-action comparisons evaluate SRNet against FCN using 2D ground-truth input with both MPJPE and PA-MPJPE, reporting Basic and Pixel normalization.
  • Cross-action results: Under the cross-action protocol, MPJPE decreases from 80.6mm to 64.3mm and PA-MPJPE from 60.5mm to 49.4mm.These correspond to relative improvements of 20.2% and 18.3%, respectively.

5 Additional Qualitative Results

Qualitative evaluations examine difficult rare poses, unseen cross-action poses, and predictions on MPI-INF-3DHP. The visualizations compare baseline predictions, SRNet outputs, and ground truth.

  • Human3.6M rare poses: On rare subject-protocol poses, SRNet predicts challenging configurations such as kowtow, side-lying, and legs lifting well.
  • Cross-action generalization: Under the cross-action protocol, Figure 2 compares FCN and SRNet on unseen test actions, including sit and sit down.The examples train on greet or take photos before testing on the unseen actions.
  • MPI-INF-3DHP: On MPI-INF-3DHP, Figure 3 presents baseline and SRNet predictions against ground truth from front and top viewpoints.
Loading 2007.09389v1…