Estimating Egocentric 3D Human Pose in Global Space

1. Introduction

• Mo2Cap2 and xR-egopose estimate local rather than global poses, while direct SLAM-based projection produces jitters, tracking failures, and unrealistic motion.Local poses alone are insufficient for applications such as animating avatar locomotion in xR environments.Source: §1 · pp. 1–2
• A spatio-temporal optimization framework uses CNN-detected keypoints and local and global motion priors from an efficient convolutional VAE to produce accurate, stable poses.The convolutional VAE addresses the optimization inefficiency and instability of RNN-based VAEs, which can suffer vanishing and exploding gradients.Source: §1 · p. 2
• An uncertainty-aware reprojection loss sums heatmap probabilities at projected 3D joint locations to reduce the impact of severe egocentric self-occlusion.This replaces direct comparison of projected joints with predicted 2D joint positions.Source: §1 · p. 2
• A separate global pose optimizer with its own VAE makes local body poses consistent with camera poses estimated by SLAM.The framework addresses inconsistencies between independently estimated local poses and camera poses.Source: §1 · p. 2
• The method outperforms state-of-the-art methods quantitatively and qualitatively, improving global and local pose accuracy over various baselines.It was evaluated on the Mo2Cap2 dataset and a new benchmark collected with two subjects performing various motions.Source: §1 · p. 2

Egocentric 3D full body pose estimation

• Unlike prior egocentric approaches that omit temporal consistency or motion realism, this method uses motion-prior optimization to improve prediction consistency and accuracy.Earlier work includes environment-based reconstruction, multi-camera capture, and single-frame monocular fisheye methods.Source: §2 · p. 2

Leveraging learned prior in 3D pose estimation

• Prior pose methods learn mocap priors with Gaussian models, VAEs, normalizing flows, or GANs, while this method learns a global motion prior with a lightweight sequential VAE for direct global-coordinate optimization.Examples include Gaussian priors for SMPL [3], [2], VAE priors [34], normalizing flows [49], and adversarial methods [46], [14] [15], [51], [19].Source: §2 · pp. 2–3

Monocular 3D pose estimation in video

• Unlike monocular video methods using filtering, recurrent networks, temporal convolutions, or graph convolutions, this method uses a 1D-convolution sequential VAE within its optimization framework.Single-image approaches—including 2D-to-3D lifting [6] [12] [29] and direct 3D regression [30] [41] [42] [20]—can exhibit temporal jitter.Source: §2 · p. 3

3. Method

• For each 10-frame segment, the pipeline estimates local poses and heatmaps, optimizes local motion, transforms poses using SLAM camera estimates, then refines global poses with a learned global prior.Source: §3 · p. 3

3.1. Local Pose Estimation

• The local estimator supplies 15-joint 3D poses and 2D heatmaps; the framework supports any egocentric estimator and evaluates Mo2Cap2 and xR-egopose.Source: §3.1 · p. 3

3.2. Local Pose Optimization

• Local pose estimates suffer from monocular depth ambiguity and egocentric occlusion, so the method adds spatio-temporal optimization using a learned pose prior, reprojection, and regularization.Source: §3.2 · p. 4

3.2.1 Learning Motion Prior

• A sequential VAE trained on AMASS local-motion sequences learns a latent prior; its 5-layer 1D convolutional encoder and decoder parallelize over time and are described as more stable than RNNs.Source: §3.2.1 · p. 4

3.2.2 Optimizing Local Poses with Local Motion Prior

• Local pose optimization searches the learned latent space using heatmap reprojection plus pose, temporal-smoothness, and bone-length regularizers, maximizing predicted heatmap values at projected joints.Source: §3.2.2 · p. 4

3.3. Global Pose Estimation

• Because optimized local poses may be inconsistent with camera poses, the method refines SLAM-transformed initial global poses using a second spatio-temporal optimization and an AMASS-trained global-pose prior.Source: §3.3 · p. 5, §3.3.2 · p. 5

3.3.1 Initialization

• ORB-SLAM2 [33] estimates camera poses after a fixed square body mask removes most person-region features, and those poses transform local joints into initial world-space poses.Source: §3.3.1 · p. 5

3.3.2 Global Pose Optimizer

• The global optimizer uses a learned global-motion latent prior and pose, temporal-smoothness, and bone-length terms; its example alleviates the initial footskate artifact.Source: §3.3.2 · p. 5, Figure 4 · p. 5

4.1. Datasets

• Local pose estimators are trained on synthetic Mo2Cap2 data and sequential VAEs on AMASS; evaluation includes Mo2Cap2 and a new egocentric dataset.The new dataset uses a head-mounted fisheye camera and multi-view motion-capture ground truth, with around 12,000 frames of two actors performing 13 action types.Source: §4.1 · p. 6

4.2. Evaluation Metrics

• Evaluation uses PA-MPJPE for single-pose accuracy, BA-MPJPE after bone-length alignment, and global MPJPE for global joint-position accuracy.Global MPJPE aligns batches of 100 frames to ground truth; BA-MPJPE resizes skeletons to standard bone lengths before Procrustes alignment.Source: §4.2 · p. 6

4.3. Comparison with State-of-the-art Results

• The method outperforms Mo2Cap2 [45] and xR-egopose [43] across all three metrics; qualitative results also demonstrate global poses projected onto an external camera.The comparisons include Mo2Cap2 and the new test dataset; the paper reports significant accuracy improvements and illustrates global pose estimates from a third-view camera.Source: Table 1 · p. 6, §4.3 · p. 6, Figure 6 · p. 7

4.4. Ablation Study

• Both local and global pose optimizers contribute: heatmap reprojection constrains poses to 2D predictions, while the global VAE prior improves global and local MPJPEs.Removing either optimizer worsens the results; the ablation also identifies the VAE prior as aligning limb movement with the global camera pose.Source: §4.4 · p. 7, Table 2 · p. 7
• Motion priors correct ambiguous poses under severe occlusion and produce smoother, less ambiguous predictions than optimization without a prior.The paper illustrates a severely occluded leg: without the prior, the pose error remains large, whereas the prior corrects the estimated pose.Source: §4.4 · p. 7, Table 2 · p. 7, Figure 7 · p. 8
• The sequential VAE performs better than Gaussian-mixture and single-frame VAE priors, which are less effective at capturing pose-sequence structure over time.The authors attribute the GMM gap to the VAE's neural-network feature extractor and note that a single-frame VAE cannot capture temporal priors.Source: §4.4 · pp. 7–8, Table 2 · p. 7
• CNN-based sequential VAEs outperform VIBE, MEVA, and MLP-based alternatives in optimization accuracy, with more stable optimization than RNN-based VAEs.Optimization takes 195.7 ms per 10-frame segment for this method, versus 552.1 ms for VIBE and 1139.4 ms for MEVA; RNN-based VAEs are less stable due to gradient explosion.Source: Table 2 · p. 7, §4.4 · p. 8, Figure 8 · p. 8
• The heatmap reprojection error handles uncertain 2D joint predictions better than conventional reprojection error, improving the resulting pose.In the qualitative example, the estimated right-hand position is wrong, but the ground-truth hand remains covered by the heatmap.Source: Table 2 · p. 7, §4.4 · p. 8, Figure 9 · p. 8

• Extensive experiments show that the method outperforms state-of-the-art methods, and the effects of its individual components are evaluated.Source: §5 · p. 8
• The method estimates global poses with a head-mounted fisheye camera using spatio-temporal optimization with a sequential VAE motion prior, a global motion prior, and heatmap-based reprojection error.Source: §5 · p. 8
• Future work will study integrating depth sensors, using optimized real-world 3D poses to finetune the local pose estimation network, and extending the method to multi-person scenarios.Source: §5 · p. 8