PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera

Denis Tome, Thiemo Alldieck, Patrick Peluse, Gerard Pons-Moll, Lourdes Agapito, Hernan Badino, Fernando De la Torre

arXiv:2011.01519v1cs.CV

TL;DR

Downward-looking HMD cameras produce egocentric images with severe self-occlusion and perspective-driven differences in body resolution, complicating 3D pose estimation. The paper combines an uncertainty-aware multi-branch pose architecture with a photo-realistic synthetic dataset. It reports improvements over egocentric methods and second-best performance on Human3.6M, while noting failures under extreme occlusion and when hands leave the camera's field of view.

  • Problem

    Downward-looking HMD cameras create severe self-occlusion and perspective effects with large upper-to-lower-body resolution differences, making egocentric 3D pose estimation challenging.

    Source: §1 · p. 1

  • Method

    The paper introduces a multi-branch auto-encoder that predicts 3D pose and local joint rotations while reconstructing input heatmaps to preserve uncertainty information.

    Source: §4 · p. 4, §4.2 · p. 5

  • Results

    The method reports more than 25% improvement over Mo2Cap2 on synthetic and real-world benchmarks and second-best performance after on Human3.6M.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The paper concludes that its architecture supports accurate pose reconstruction and avatar driving, and generalizes to front-facing 3D pose estimation with state-of-the-art performance.

    Source: §6 · p. 9

    Failures occur under extreme occlusion and when hands are outside the field of view; adding cameras for wider field of view and multi-view sensing is future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

We present a solution to egocentric 3D body pose estimation from monocular images captured from downward looking fish-eye cameras installed on the rim of a head mounted VR device. This unusual viewpoint leads to images with unique visual appearance, with severe self-occlusions and perspective distortions that result in drastic differences in resolution between lower and upper body. We propose an encoder-decoder architecture with a novel multi-branch decoder designed to account for the varying uncertainty in 2D predictions. The quantitative evaluation, on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric approaches. To tackle the lack of labelled data we also introduced a large photo-realistic synthetic dataset. xR-EgoPose offers high quality renderings of people with diverse skintones, body shapes and clothing, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of theart results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.

1 INTRODUCTION

Egocentric pose estimation from a downward-looking HMD camera is difficult because of perspective distortion, uneven resolution, and self-occlusion. The paper addresses these challenges with an uncertainty-aware multi-branch architecture and reports improvements over egocentric and front-facing pose-estimation baselines.

1 INTRODUCTION

  • The HMD-mounted downward-looking fish-eye camera produces strong perspective distortion, large upper-to-lower-body resolution differences, and lower-body self-occlusion.The figure contrasts external and egocentric views and marks occluded body parts and pixel density.Source: §1 · p. 1, Figure 2 · p. 2
  • The proposed architecture encodes differing uncertainty across upper- and lower-body joints arising from varying resolution, perspective effects, and self-occlusions.The paper presents this architecture alongside a new training corpus to address the unusual egocentric appearance.Source: §1 · p. 1
  • More than 25% improvement over Mo2Cap2 is reported on synthetic and real-world benchmarks, while the method ranks second after on Human3.6M.The evaluation uses benchmarks with ground-truth 3D annotations.Source: §1 · pp. 1–2
  • The multi-branch decoder reconstructs 2D heatmaps and predicts rotations, and ablations attribute drastic 3D pose improvements to its introduction.The branches also provide tools to control uncertainty in the latent space.Source: §1 · p. 2

2 RELATED WORK

Related work spans external-camera 3D pose estimation, egocentric capture, and wearable sensing. The paper builds on CNN-based methods while extending its earlier approach with architecture analysis and per-part rotation prediction.

Monocular 3D Pose Estimation from an External Camera

  • External-camera methods use direct 3D regression or 2D detection followed by 3D lifting, with later work combining tasks or using weakly and self-supervised losses.Limited in-the-wild 3D annotations have also motivated photo-realistic synthetic datasets.Source: Monocular 3D Pose Estimation from an… · pp. 2–3

2 RELATED WORK

  • Egocentric methods have often captured only upper-body motion; prior full-body systems include chest-mounted, stereo-fisheye, and monocular head-mounted setups, each with stated accuracy or usability constraints.The paper reports that its approach outperforms Xu et al. by at least 20% on indoor and outdoor sequences.Source: Monocular 3D Pose Estimation from an… · p. 3

Monocular 3D Pose Estimation from an External Camera

  • Compared with the earlier conference paper, this journal work analyzes architectures, finds UNets outperform its ResNet for synthetic-to-real transfer, and adds per-part rotation prediction for avatar animation.The reported comparison says UNets significantly outperform the originally proposed ResNet architecture.Source: Monocular 3D Pose Estimation from an… · p. 3

2 RELATED WORK

  • Wearable IMU and limb-camera approaches offer alternatives, but are described as intrusive, difficult to calibrate, or impractical in realistic scenarios.Video-IMU fusion methods cited require line of sight with an external camera.Source: Monocular 3D Pose Estimation from an… · p. 3

3 xR-EGOPOSE SYNTHETIC DATASET

xR-EgoPose is a photo-realistic synthetic egocentric dataset designed to address the scarcity of data for evaluating HMD-based 3D pose estimation. It varies characters, environments, lighting, and actions, and provides 383K frames across training, test, and validation sets.

3 xR-EGOPOSE SYNTHETIC DATASET

  • The dataset addresses limited statistical analysis in a relatively new HMD-based egocentric 3D pose problem with a photo-realistic synthetic corpus containing ground-truth data.The paper states that, to its knowledge, only one dataset was then available for analyzing these algorithms.Source: §3 · p. 3
  • Scenes are generated from randomized characters, environments, lighting rigs, and mocap-based actions, with varied body types, skin tones, clothing, and camera placement.Character heights range from 155 cm to 189 cm.Source: §3 · p. 3
  • Each frame includes RGB, depth, normals, segmentation, pixel world positions, and metadata such as 3D joint positions and camera pose.Images are rendered at 1024 × 1024 pixels and 16-bit color depth, then downsampled to 8-bit for training and testing.Source: §3 · p. 4

3.1 Training, Test, and Validation Sets

  • The dataset contains 383K frames from 23 male and 23 female characters, partitioned into 252K training, 115K test, and 16K validation frames.The table caption specifies frame counts per action for train and test; actions not listed are validation data.Source: §3.1 · p. 4, Table 1 · p. 4

4 ARCHITECTURE

The architecture separates 2D joint detection from 3D pose regression, using a multi-branch auto-encoder to preserve uncertainty in the heatmap representation. Its auxiliary branches predict local rotations and reconstruct heatmaps, while the training setup supports independently trained modules and mixed 2D/3D supervision.

4 ARCHITECTURE

  • The two-step pipeline detects 2D joint heatmaps from RGB images, then regresses 3D joint coordinates, local rotations, and reconstructed heatmaps with a multi-branch auto-encoder.The modules can be pretrained independently and fine-tuned end-to-end; the auxiliary branches are training-only and can be removed at test time.Source: §4 · p. 4

4.1 2D Pose Detection

  • The 2D detector maps 368×368×3 RGB images to 47×47×15 joint heatmaps, and its training loss is mean squared error against ground-truth heatmaps.The authors experimented with ResNet 50 and U-Net architectures.Source: §4.1 · p. 4

4.2 2D-to-3D Mapping

  • The 3D module regresses pose from heatmaps and preserves their uncertainty in the pose embedding through branches for 3D pose, local rotations, and heatmap reconstruction.The rotation branch uses quaternion representations for training stability and is associated with smoother transitions across frame-by-frame pose estimates.Source: §4.2 · p. 5

4.3 Training Details

  • The loss can use mixed 2D- and 3D-annotated data: samples with only 2D annotations contribute through the heatmap loss alone.Training used three epochs, a learning rate of 1e−3, and mini-batches of 16; loss weights were selected by grid search.Source: §4.2 · p. 5, §4.3 · pp. 5–6

5 EXPERIMENTAL EVALUATION

The evaluation tests the method on synthetic and real egocentric data, front-facing Human3.6M, and mixed 2D/3D supervision. Across these settings, the experiments report improved reconstruction, robustness, or generalization under the conditions tested.

5 EXPERIMENTAL EVALUATION

  • The evaluation uses MPJPE unless otherwise stated and establishes a reproducible xR-EgoPose baseline with randomly initialized ResNet 50.The authors avoid pretraining for that baseline because it affects final results.Source: §5 · p. 6

5.1 Evaluation on our Egocentric Synthetic Dataset

  • On xR-EgoPose, the approach reduces MPJPE over Martinez et al. by 52.3% overall, 36.4% for the upper body, and 60% for the lower body.The comparison retrains Martinez et al. on the same synthetic training set; branch ablations report 130.4 mm for the single-branch encoder versus 54.7 mm for the full multi-branch model.Source: §5.1 · p. 6

5.2 Character Animation Using Estimated Rotations

  • The rotation branch supports character animation, with plausible rotations and temporal consistency even when limbs leave the camera’s field of view.Frame-by-frame predictions remain generally smooth, although limited jitter is reported.Source: §5.2 · pp. 6–7

5.3 Heatmap Estimation: Architecture Ablation

  • In experiments, a U-Net 2D pose estimator improved the pipeline’s MPJPE to 41.0 mm and, despite training only on synthetic data, predicted plausible heatmaps on unseen real data; its error increased slowly under white Gaussian image noise.A pre-trained ResNet 50 achieved 51.1 mm MPJPE, compared with 54.7 mm for random initialization.Source: §5.3 · p. 7

5.4 Lifting Network: Parameter Ablation

  • Across heatmap-estimation networks, a 50-dimensional pose embedding gives the best results; smaller embeddings raise errors and larger ones slightly impair them.Regressing the full heatmap performs best among the tested heatmap dimensions.Source: §5.4 · p. 8, Table 5 · p. 9

5.5 Evaluation on Egocentric Real Datasets

  • On real egocentric data, the dual-branch approach substantially outperforms Mo2Cap2 indoors and outdoors, while the controlled indoor comparison is nearly tied.The model using an ImageNet-pretrained U-Net gives the best results; the small xR-EgoPoseR test set supports qualitative, not meaningful numerical, evaluation.Source: §5.5 · pp. 8–9

5.6 Evaluation on Front-facing Cameras

  • On Human3.6M, the approach is on par with state-of-the-art methods and ranks second among non-temporal methods.Evaluation uses two protocols, with MPJPE for Protocol 1 and Procrustes-aligned MPJPE for Protocol 2.Source: §5.6 · p. 9

5.7 Mixing 2D and 3D Ground Truth Datasets

  • Adding 2D-only annotations improves 3D pose estimates in both egocentric and front-facing settings.The study compares xR-EgoPose subsets with the same amount of 3D labels but more 2D-labeled images, and reports improvements on Human3.6M when adding COCO and MPII annotations.Source: §5.7 · p. 9

6 CONCLUSION

The conclusion presents the pose-estimation architecture and xR-EgoPose synthetic dataset as contributions, reports improved performance on egocentric and front-facing benchmarks, and notes failures under extreme occlusion and when hands leave the camera’s field of view.

6 CONCLUSION

  • The architecture reports over 24% accuracy improvement on competitor datasets and state-of-the-art performance for front-facing-camera 3D pose estimation.The conclusion also says the architecture can drive a virtual avatar from its pose estimates.Source: §6 · p. 9
  • The authors introduce xR-EgoPose, a large-scale photorealistic synthetic dataset described as essential for training.The supplied passage ends before specifying its availability details.Source: §6 · p. 9
  • Even little uncertainty information has a dramatic impact on reconstruction accuracy in tests of different heatmap reconstruction sizes.Table 6 reports average per-joint reconstruction error across the entire test set.Source: Table 6 · p. 9
  • More than 21.6% (13.24 mm) on indoor data and more than 25.4% (20.45 mm) on outdoor data: the approach outperforms all competitors using only synthetic training data.With a pre-trained U-Net, the improvements are 24.9% (14.79 mm) indoors and 32.28% (26.03 mm) outdoors.Source: Table 8 · p. 10
  • Failures remain under extreme occlusion, and hands outside the camera’s field of view cannot be measured; additional cameras are proposed as future work.The stated future direction is multi-view sensing to cover more of the field of view.Source: §6 · p. 10

Related papers

Loading 2011.01519v1…