Source-linked AI summary
EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices
Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, Siyu Tang
TL;DR
Egocentric estimation of an interaction partner’s 3D body is hindered by limited datasets and challenging HMD imagery. EgoBody addresses this gap with a large-scale multimodal dataset, calibrated 3D reconstruction, and a benchmark for interactee pose and shape estimation. Existing methods perform substantially worse in the egocentric setting, while fine-tuning on EgoBody improves their performance and generalizes to another dataset.
Problem
Research on estimating an interaction partner’s 3D pose, shape, and motion from egocentric views is limited by inadequate dataset scale, modalities, ground-truth quality, and interaction diversity.
Method
EgoBody records multimodal HoloLens2 egocentric data and fits SMPL-X body meshes to calibrated multi-view RGB-D data for interacting subjects in diverse scenarios.
Results
EgoBody establishes the first egocentric interactee 3DHPS benchmark; existing methods perform worse on EgoBody than on third-person data, while EgoBody fine-tuning improves performance and transfers to You2Me.
Takeaways & Limitations
EgoBody provides a basis for evaluating and improving egocentric human pose and shape estimation during social interactions.
Takeaways & Limitations
Synchronization between HoloLens2 and Kinect has small temporal misalignment, especially observable during fast motions.
Abstract
from arXiv · showhide
Understanding social interactions from egocentric views is crucial for many applications, ranging from assistive robotics to AR/VR. Key to reasoning about interactions is to understand the body pose and motion of the interaction partner from the egocentric view. However, research in this area is severely hindered by the lack of datasets. Existing datasets are limited in terms of either size, capture/annotation modalities, ground-truth quality, or interaction diversity. We fill this gap by proposing EgoBody, a novel large-scale dataset for human pose, shape and motion estimation from egocentric views, during interactions in complex 3D scenes. We employ Microsoft HoloLens2 headsets to record rich egocentric data streams (including RGB, depth, eye gaze, head and hand tracking). To obtain accurate 3D ground truth, we calibrate the headset with a multi-Kinect rig and fit expressive SMPL-X body meshes to multi-view RGB-D frames, reconstructing 3D human shapes and poses relative to the scene, over time. We collect 125 sequences, spanning diverse interaction scenarios, and propose the first benchmark for 3D full-body pose and shape estimation of the social partner from egocentric views. We extensively evaluate state-of-the-art methods, highlight their limitations in the egocentric scenario, and address such limitations leveraging our high-quality annotations. Data and code are available at https://sanweiliti.github.io/egobody/egobody.html.
1 Introduction
EgoBody addresses the lack of data for estimating an interaction partner’s 3D body pose, shape, and motion from egocentric views. It provides multimodal egocentric and third-person data with high-quality 3D annotations and evaluates existing methods in this challenging setting.
- Egocentric interactee estimation is important for interaction understanding in applications such as assistive robotics and AR/VR.
- Existing methods face severe body truncations, motion blur, and people entering or leaving the field of view in HMD data.
- Existing datasets lack combinations of egocentric views, interactions, 3D scene context, body shape, and 3D ground truth.
- EgoBody provides a large-scale egocentric dataset with multimodal HoloLens2 streams, including RGB, depth, head, hand, and eye-gaze tracking.
- EgoBody supplies high-quality 3D pose, shape, and motion ground truth for both interacting subjects, alongside reconstructed 3D scenes.
- The benchmark evaluates state-of-the-art 3DHPS methods, identifies egocentric limitations, and shows that fine-tuning on EgoBody improves performance.
2 Related Work
Prior datasets and methods largely emphasize third-person views, camera-wearer pose, or coarse interaction labels. EgoBody is distinguished by combining calibrated egocentric and third-person views, diverse interactions, multimodal data, and rich 3D ground truth.
- Third-person datasets provide substantial 3D pose and motion data but are often limited to constrained studio setups and marker-contaminated images.
- Many egocentric datasets focus on hand-object interaction or action recognition without 3D ground truth.
- Egocentric body datasets commonly estimate the camera wearer rather than the interactee, while EgoMoCap provides limited interaction diversity and only egocentric RGB.
- Most existing 3D pose methods require full-body images and therefore lack robustness when egocentric interactee views are occluded or truncated.
- EgoBody combines calibrated egocentric and third-person images, varied interaction scenarios, multimodal data, rich 3D ground truth, and camera-wearer eye gaze.
3 Building the EgoBody Dataset
EgoBody combines HoloLens2 egocentric sensing with synchronized multi-view Kinect capture to record diverse two-person interactions and reconstruct calibrated SMPL-X body annotations. Its pipeline refines Kinect–HoloLens2 calibration and temporally fits full-body motion using multi-view, egocentric, and scene constraints.
- Data acquisition: Each sequence involves two participants performing one or more interaction scenarios, with one participant wearing HoloLens2 and the interactee device-free.
- Interaction scenarios: Scenarios span cooperation, social exchange, conflict, conformity, and other sociology-based categories, while improvisation and human–scene interactions increase motion diversity.
- Data acquisition: HoloLens2 records synchronized RGB, depth, head, hand, and eye-gaze streams, while three to five Azure Kinects capture synchronized third-person RGB-D views.
- Ground-truth acquisition: The pipeline fits SMPL-X parameters for each subject and frame using multi-view joint and depth terms, body priors, scene contact, and collision constraints.
- Ground-truth acquisition: A keypoint-based refinement minimizes projected-joint error against egocentric OpenPose detections while regularizing deviation from the initial Kinect–HoloLens2 calibration.
- Ground-truth acquisition: Temporal fitting adds egocentric joint constraints, smoothness, priors, and friction terms to produce realistic, consistent motion, applying egocentric constraints to visible interactees.
4 EgoBody Dataset
EgoBody provides large-scale synchronized multi-view and egocentric recordings of diverse social interactions, with reconstructed bodies, scenes, and eye gaze. The dataset also supports analysis of attention and egocentric pose estimation under incomplete visibility.
- Dataset scale: 125 sequences from 36 subjects cover diverse social interactions in 15 indoor scenes.
- Dataset scale: EgoBody contains 219,731 synchronized multi-view frames with SMPL-X pose and shape annotations for both subjects and a 3D scene mesh.
- Egocentric data: The EgoSet contains 199,111 calibrated, synchronized HoloLens2 RGB frames, including 175,611 frames in which the interactee is visible.
- Eye gaze and attention: Eye-gaze reconstruction indicates that camera-wearer attention is highly focused on the interactee and tends to be closer to upper-body joints.
5 Experiments
EgoBody benchmarks egocentric 3DHPS methods, reveals substantial performance degradation and analyzes visibility-related failure modes. Fine-tuning on EgoBody improves accuracy and robustness on both EgoBody and You2Me.
- Benchmark Evaluation: The benchmark evaluates state-of-the-art 3DHPS methods on egocentric interactee images using MPJPE and V2V errors with translation-only or Procrustes alignment.Table 3 includes fitting-based, parametric regression, and nonparametric mesh methods.
- Baseline Evaluation: 77% higher 3D joint error occurs on EgoBody than on 3DPW on average, while methods show more varied performance on EgoBody.The comparison uses PA-MPJPE and indicates that third-person-view benchmarks do not sufficiently reflect egocentric difficulty.
- Visibility Analysis: Lower body visibility produces lower error overall, with PARE and EFT achieving the best results under truncation.PARE uses body-part attention, whereas EFT uses aggressive cropping as training augmentation.
- Joint Error Analysis: Wrist joints remain relatively error-prone despite good visibility, while less-visible joint groups generally exhibit higher error.The analysis identifies extremities as a persistent difficulty for existing 3DHPS models.
- Baseline Improvement: 42%/36%/18% lower MPJPE and 35%/33%/14% lower V2V are achieved by SPIN-ft/METRO-ft/EFT-ft than their original models.Improvements occur across all reported metrics after fine-tuning on EgoBody.
- Cross-dataset Evaluation: Fine-tuning also improves cross-dataset PA-MPJPE on You2Me for SPIN, EFT, and METRO, reaching 87.9, 85.6, and 88.2 mm respectively.The corresponding original-model values are 152.8, 95.8, and 117.7 mm.
6 Conclusion
The paper presents EgoBody as a multimodal dataset and benchmark for interacting people’s 3D pose and shape in egocentric settings. Its annotations support systematic evaluation, limitation analysis, and generalizable gains from fine-tuning.
- Dataset and Benchmark: EgoBody captures human pose, shape, and motion for interacting people in diverse environments using multimodal egocentric and third-person data.The dataset includes ground-truth 3D pose and shape for all interacting subjects.
- Conclusion: The benchmark exposes limitations of state-of-the-art egocentric 3DHPS methods and demonstrates significant, generalizable gains using EgoBody annotations.The conclusion identifies improved understanding of egocentric human motions, behaviors, and social interactions as a potential direction.
**Supplementary Material**
The supplementary material describes calibration refinement and the SMPL-X representation used for reconstructing human bodies in 3D.
- Calibration: Calibration begins with checkerboard-based initialization, then refines Kinect registrations with ICP on reconstructed scene point clouds.Kinect-HoloLens2 calibration receives an additional keypoint-based optimization refinement.
- Body Representation: SMPL-X maps global translation, body shape, pose, and facial expression parameters to a 10,475-vertex triangle body mesh.Its pose parameters include body, facial, and hand poses, with joints defined in a neutral pose before posing.
A.3 Data Processing
The data-processing pipeline cleans detections, filters unreliable joints, and evaluates reconstructed motion quality against annotations and AMASS.
- Detection and Cleaning: Human point clouds are extracted from Kinect depth using refined instance masks, while failed detections and inaccurate self-occluded joints are manually cleaned.Depth differences from the target person’s median joint depth help filter unreliable 2D joints.
- Subset Selection: Extreme body truncations and spurious detections are excluded when constructing the EgoSet-interactee subset.OpenPose detections are manually reviewed and body fragments from the same person may be merged.
- Ground-truth Evaluation: 31.08 pixels is the mean 2D joint error on 1,517 manually annotated 1920 × 1080 frames.The error compares projected reconstructed joints with Amazon Mechanical Turk annotations.
- Motion Quality: Lower PSKL indicates motions more similar to the natural motions in AMASS, providing the basis for motion-smoothness evaluation.The dataset’s motion distribution is compared with AMASS using joint-acceleration distributions.
C More Statistics
EgoBody statistics characterize visibility, gaze, and interaction conditions in the dataset. Lower-body joints are often invisible, while attention tends to focus near upper-body joints across varied interaction distances and image sharpness.
- Joint invisibility: Over 60% of body joints are invisible in extreme frames, with knees invisible around 50% and feet around 80% of frames.Neck, shoulder, spine, and elbow joints are most visible; wrists and head have around 10% invisibility.
- Eye gaze and attention: The eye gaze point is generally closer to upper-body joints during interactions.The analysis projects the 3D gaze point into egocentric images and measures distances to body joints.
- Interaction distance: Interaction distances between subjects range from 0.90m to 3.48m based on pelvis-joint separation.EgoBody therefore spans a broad range of indoor two-person interaction distances.
- Motion blur: Image sharpness is quantified with the variance of the Laplacian, where higher scores indicate sharper images.The sharpness distribution is reported for the EgoSet-interactee test set.
D.1 Discussion on Evaluation Metrics
The benchmark supplements sparse joint-based evaluation with dense mesh-based error measurement and reports results under alignment choices that preserve global-orientation errors by default. Supplementary analyses document qualitative behavior and cross-dataset evaluation constraints.
- Evaluation metrics: V2V error complements MPJPE by measuring shape and denser pose discrepancies, including errors from longitudinal joint rotations.MPJPE does not penalize wrong joint twisting because it evaluates only sparse joint locations.
- Qualitative analysis: Fast motions can produce body-mesh and HoloLens2-image misalignments in the supplementary qualitative analysis.The figure specifically illustrates misalignment during rapid hand movements.
- Alignment choices: Translation-only alignment is the default because Procrustes Alignment removes global-orientation discrepancies, a major error source for most methods.PA aligns scale, translation, and rotation, whereas translation-only alignment aligns bodies at the pelvis.
- Supplementary evaluations: The supplementary material includes qualitative baseline results and evaluates methods on the You2Me egocentric dataset.You2Me provides chest-mounted GoPro images and world-coordinate ground-truth joints, but unknown camera calibration prevents translation-only MPJPE.
D.4 Experiment with Motion Blur Augmentation
Additional motion-blur augmentation on the original training data does not resolve EgoBody’s egocentric challenges. Fine-tuning on EgoBody remains more effective, while the supplementary material also documents annotation procedures and related implementation context.
- Comparison with dataset fine-tuning: A clear performance gap remains between original EFT and models fine-tuned on EgoBody, despite EFT using aggressive image-cr data augmentation.The result indicates that augmentation on the original datasets does not fully address EgoBody’s egocentric challenges.
- Motion blur augmentation: No improvements over original SPIN are observed when adding motion-blur augmentation during fine-tuning on its original training data.Blur direction, angle, and kernel size are randomized with probability 0.5 across multiple kernel settings.
- Compared methods: METRO directly estimates body vertices and 3D joints using a transformer encoder that models vertex-vertex and vertex-joint interactions.The method is described as a model-free formulation in the supplementary implementation discussion.
- AMT annotation: Left-right annotation flips are corrected by comparing annotations with and without side reversal and applying outlier detection.This procedure addresses non-negligible noise caused by users reversing body sides.
F Limitations
Synchronizing HoloLens2 and Kinect data requires software alignment because hardware synchronization is unavailable. Small temporal misalignment remains observable during fast motions, although reconstruction accuracy remains high.
- Synchronization limitation: Small temporal misalignment remains during fast motions because HoloLens2 and Kinect cannot be synchronized through hardware.The clocks are aligned using a flashlight signal and HoloLens2 timestamps, but rapid hand movements can still reveal misalignment.
- Synchronization limitation: Despite this limitation, the authors report high reconstruction accuracy based on their reconstruction-accuracy evaluation.The limitation is therefore a synchronization boundary rather than a reported failure of reconstruction accuracy.