PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Camera Viewpoint Improves Cross-dataset Generalization for 3D Human Pose Estimation

Zhe Wang, Daeyun Shin, Charless C. Fowlkes

arXiv:2004.03143v1cs.CV

TL;DR

Monocular 3D pose methods face limited training-data diversity and uncertain generalization beyond their source datasets. The paper analyzes five datasets, focuses on camera viewpoints relative to body-centered coordinates, and adds viewpoint prediction alongside pose. Joint prediction systematically improves cross-dataset generalization and achieves state-of-the-art performance across the five datasets.

  • Problem

    Limited training-data diversity leaves it unclear how well monocular 3D pose methods generalize outside their training datasets.

  • Method

    The paper analyzes viewpoint and pose biases across five datasets and predicts camera viewpoint relative to a torso-defined body-centered coordinate frame as an auxiliary task.

  • Results

    Joint viewpoint-and-pose prediction systematically improves cross-dataset generalization and achieves state-of-the-art performance across five datasets.

  • Takeaways & Limitations

    Camera viewpoint is a useful supervisory signal for reducing dataset bias and improving robustness in cross-dataset 3D human pose evaluation.

Abstract

from arXiv · show

Monocular estimation of 3d human pose has attracted increased attention with the availability of large ground-truth motion capture datasets. However, the diversity of training data available is limited and it is not clear to what extent methods generalize outside the specific datasets they are trained on. In this work we carry out a systematic study of the diversity and biases present in specific datasets and its effect on cross-dataset generalization across a compendium of 5 pose datasets. We specifically focus on systematic differences in the distribution of camera viewpoints relative to a body-centered coordinate frame. Based on this observation, we propose an auxiliary task of predicting the camera viewpoint in addition to pose. We find that models trained to jointly predict viewpoint and pose systematically show significantly improved cross-dataset generalization.

1 Introduction

The paper studies dataset bias in monocular 3D human pose estimation and proposes camera-viewpoint prediction as an auxiliary task to improve cross-dataset generalization.

  • 1 Introduction: The five pose datasets differ in capture systems, cameras, environments, camera viewpoints, pose distributions, body sizes, and camera parameters.These systematic differences and their effects on performance had not been carefully analyzed.
  • 1 Introduction: The proposed auxiliary task predicts the relative rotation between camera viewing direction and a torso-defined body-centered coordinate system.Its ground truth can be derived from existing 3D pose datasets without additional labels.
  • 1 Introduction: Viewpoint prediction systematically improves cross-dataset pose generalization and yields state-of-the-art performance across five datasets.The authors report improvements over two popular baselines and state-of-the-art performance on five datasets.
  • 1 Introduction: The paper analyzes viewpoint and body-centered-pose distributions across contemporary 3D human pose datasets.This analysis targets dataset bias and cross-dataset evaluation.

2 Related Work

Related work covers cross-dataset evaluation, coordinate-frame choices, viewpoint-invariant prediction, 3D pose estimation, and multi-task training. The paper differs by using body-centered coordinates only as an auxiliary training signal for camera-centered pose.

  • Cross-Dataset Generalization and Evaluation: Prior 3D pose methods largely train and evaluate on a small set of large public datasets, with limited cross-dataset evaluation.Existing work spans datasets including H36M, 3DHP, TOTALCAPTURE, 3DPW, and GPA.
  • Cross-Dataset Generalization and Evaluation: Dataset-bias studies in object recognition and depth estimation motivate analyzing systematic differences rather than assuming datasets are interchangeable.Prior work proposed comparative dataset studies and scale-invariant objectives for mixing otherwise incompatible datasets.
  • Coordinate Frames for 3D Human Pose: Standard pose targets express joints relative to a root in camera coordinates, while this paper explores a body-centered coordinate system rotated relative to the camera frame.The authors argue that camera viewpoint is an overlooked component of experimental design.
  • Coordinate Frames for 3D Human Pose: Viewpoint-invariant prediction has precedent in 3D object and hand pose estimation, but canonical pose prediction is difficult for highly articulated objects.This paper instead uses body-centered coordinates as an auxiliary task that improves camera-centered pose prediction.
  • Multi-task Training: Multi-task training commonly combines related prediction tasks to improve model accuracy or computation.The cited examples include detection, classification, segmentation, surface normals, depth, keypoints, and viewpoint.

3 Variation in 3D Human Pose Datasets

The study separates camera viewpoint from body-centered pose to characterize variation and bias across five 3D human pose datasets. Camera viewpoints show substantial dataset-specific differences, while body-centered pose distributions overlap considerably after viewpoint and body-size normalization.

  • Body-centered Coordinate Frames: The preprocessing typically removes effects of camera intrinsics and distance but does not address camera orientation, so pose is factored into viewpoint and body-centered pose.The body-centered frame is defined from the pelvis and shoulder geometry, with front, up, and right directions forming an orthogonal frame.
  • Distribution of Camera Viewpoints: Camera viewpoints differ substantially across H36M, GPA, SURREAL, 3DPW, and 3DHP, motivating their explicit separation from body-centered pose.The analysis samples 50k poses per dataset and examines azimuth and elevation relative to the human subject.
  • Distribution of Camera Viewpoints: H36M has four azimuth peaks and elevation above 0, GPA spans a broader azimuth range, and SURREAL is uniform over azimuth and elevation.H36M peaks occur near −30°, 30°, −160°, and 160°, whereas GPA views mostly range from −60° to 90°.
  • Distribution of Camera Viewpoints: 3DHP is the most azimuthally uniform nonsynthetic dataset and includes wider positive elevations because cameras were mounted at multiple heights, including the ceiling.Its camera locations also appear as dense clusters at specific azimuths in the joint camera-view distribution.
  • Distribution of Pose: After viewpoint and body-size normalization, body-centered pose distributions substantially overlap across datasets, and a dataset classifier achieves only 20% average test accuracy.The pose representation uses 14 joints common to all datasets, scaled to a common skeleton size and embedded with UMAP.

4 Learning Pose and Viewpoint Prediction

The model regularizes camera-centered 3D pose estimation by adding viewpoint prediction, representing camera-to-body rotations with quantized quaternion clusters and jointly optimizing pose and viewpoint losses.

  • Viewpoint prediction regularizes standard camera-centered pose estimation to overcome cross-dataset viewpoint biases.
  • The baseline uses a ResNet feature extractor, deconvolutional upsampling, 3D joint heatmaps, and soft-argmax coordinate extraction.
  • The added viewpoint branch predicts camera direction relative to the body-centered frame using quantized rotation clusters, with k-means clustering over quaternions.
  • Classification, regression, and combined viewpoint objectives are considered; classification produces less accurate viewpoint coordinates but the largest pose improvements.
  • The final training objective combines the negative-log-likelihood viewpoint loss with the pose loss as L = λLq + Lpose.

5 Experiments

Experiments evaluate cross-dataset generalization across five datasets using MPJPE and compare viewpoint-supervised models with baselines and state-of-the-art methods. Viewpoint supervision reduces cross-dataset error, generalizes across architectures, and achieves state-of-the-art MPJPE across all five datasets.

  • 5.1 Cross-dataset evaluation: 10.6 mm lower total cross-dataset MPJPE results from quaternion supervision when training on H36M, compared with 1.2 mm lower same-dataset error.The largest reduction occurs on GPA, at 6.9 mm.
  • 5.1 Cross-dataset evaluation: 18.6 mm lower total cross-dataset MPJPE results from quaternion supervision when training on GPA, with the largest reduction on H36M at 8.0 mm.The authors attribute this to GPA’s camera-viewing-direction bias relative to H36M.
  • 5.1 Cross-dataset evaluation: Quaternion supervision reduces cross-dataset MPJPE by 9.1 mm for SURREAL and 13.1 mm for 3DPW, while viewpoint bias explains differing gains.SURREAL’s viewpoint distribution is already relatively uniform, whereas 3DPW is biased toward azimuths from −30 to 60 degrees.
  • 5.1 Cross-dataset evaluation: 20.4 mm lower total cross-dataset MPJPE results from quaternion supervision when training on 3DHP, versus 1.5 mm lower same-dataset error.Error reductions are larger where the test viewpoint distribution has fewer views than the 3DHP training distribution.
  • 5.2 Effect of Model Architecture and Loss Functions: The viewpoint branch improves generalization when added to Zhou et al.’s different architecture, while classification-based viewpoint loss outperforms regression by 4.3 mm.Adding a canonical-pose prediction branch provides no improvement over the baseline camera-centered pose predictions.
  • 5.3 Comparison with state-of-the-art performance: State-of-the-art MPJPE is achieved on H36M, GPA, SURREAL, 3DPW, and 3DHP across the five-dataset comparison.The comparison covers both MPJPE and PCK3D, although performs slightly better on 3DHP PCK3D using additional training data.

6 Conclusions

The paper identifies dataset-specific camera-viewpoint biases and uses body-centered coordinates to provide auxiliary supervision. This reduces cross-dataset 3D pose error, improves generalization, and yields state-of-the-art same-dataset performance.

  • 6 Conclusions: Body-centered camera-viewpoint supervision significantly reduces 3D joint prediction error in cross-dataset evaluation.The model also achieves state-of-the-art performance on same-dataset evaluations.

Appendix

The appendix extends the experiments with visualizations, additional metrics, model and loss analyses, and qualitative cross-dataset evaluations. It also examines training and testing across individual and multiple datasets.

  • Appendix: Appendix experiments include UMAP visualizations, PMPJPE and PCK3D results, quaternion-loss analyses, and qualitative cross-dataset comparisons.The appendix also reports training on each single dataset and training on five datasets while testing on images from one dataset.

A UMAP Visualization

UMAP visualizations compare view-dependent and body-centered pose representations across five datasets. Body-centered, size-normalized poses overlap more across datasets, whereas root-relative representations retain distinguishable dataset structure.

  • A UMAP Visualization: Body-centered, size-normalized pose distributions show much higher overlap across datasets than view-dependent root-relative coordinates.The root-relative representation implicitly encodes camera orientation and therefore preserves distinguishable dataset structure.

B PMPJPE, PCK3D results on posenet [21] and MPJPE results on Zhou [50]

The experiments evaluate quaternion-loss augmentation for PoseNet using PMPJPE and PCK3D, and test its generality by retraining another model with the same viewpoint-prediction loss.

  • B PMPJPE, PCK3D results on posenet [21] and MPJPE results on Zhou [50]: Tables 6 and 7 evaluate the proposed quaternion loss on PoseNet using cross-dataset Procrustes-aligned MPJPE and PCK3D.The tables report baseline errors or accuracies alongside changes from adding the quaternion loss.
  • B PMPJPE, PCK3D results on posenet [21] and MPJPE results on Zhou [50]: The quaternion visualization uses dataset-specific colors and plots cluster centers in azimuth-elevation space to characterize viewpoint distributions.The quaternions represent rotations between body-centered and camera-centered frames.

D Sampled images from five datasets

The paper samples images to expose viewpoint patterns across five datasets, including directional biases, uniform coverage, and extreme elevations.

  • D Sampled images from five datasets: H36M samples reveal distinct facing-right and facing-left image groups associated with azimuth-elevation patterns.The sampled-image indices correspond to positions around a central figure.
  • D Sampled images from five datasets: GPA and SURREAL samples sweep uniformly through azimuth with randomized elevation, showing rotations from back-facing through camera-facing and back again.The visualized sequence progresses from facing backward toward the right, toward the camera, and back again.
  • D Sampled images from five datasets: 3DHP samples cover uniform azimuth, uniform elevation, and camera-centered viewpoints, with randomness added around sampled camera centers.The sampling varies azimuth and elevation separately as well as jointly around camera centers.
  • D Sampled images from five datasets: 3DPW samples include extreme-elevation views and randomly selected views.These two sampling patterns are shown separately in the corresponding figure.

E Qualitative Results

Qualitative evaluations visualize predictions across five datasets under cross-dataset training and testing conditions, while viewpoint distributions and body-centered coordinates provide the comparison framework.

  • E Qualitative Results: Models trained on GPA, SURREAL, 3DPW, and 3DHP are evaluated qualitatively across all five datasets, with images ordered by increasing MPJPE.The visualizations overlay 2D predictions on cropped images and show 3D predictions in the proposed body-centered coordinates.
  • E Qualitative Results: The viewpoint-distribution figure relates the body-centered and camera-centered coordinate frames and overlays five datasets' quaternion samples with k-means cluster centers.The clusters summarize rotations between the two coordinate systems.
  • E Qualitative Results: Testing all five trained models on the same images from each dataset enables direct qualitative comparison of training-dataset effects.Figures 18–22 repeat this comparison for H36M, GPA, SURREAL, 3DPW, and 3DHP.
  • E Qualitative Results: The qualitative figures show 2D joint predictions overlaid on RGB images and, in the cross-dataset visualizations, compare red 3D predictions with blue ground truth.The GPA-trained visualization ranks examples from left to right by increasing MPJPE.
Loading 2004.03143v1…