Source-linked AI summary
Ego-Exo4D Human Meshes Dataset: 4D Human Motion Reconstruction for Ego-Exo Captures
Abhiram Maddukuri, Georgios Pavlakos
TL;DR
Ego-Exo4D provides synchronized egocentric and multi-view exocentric video, but its 3D annotations are sparse and reconstruction at this scale is challenging. This work releases Ego-Exo4D-HM, a dataset of recovered SMPL-H motion and its reconstruction pipeline. It reports global keypoint errors and more robust reconstructions than MAMMA on challenging views.
Problem
Ego-Exo4D has synchronized egocentric and multi-view exocentric video, but its 3D annotations are sparse and current reconstruction systems can be brittle at this scale.
Source: §1 · p. 1Method
An optimization-based pipeline combines synchronized multi-view keypoints with Ego-Exo4D camera calibrations to recover world-frame SMPL-H motion.
Source: §1 · p. 1, §3.2 · p. 4, §3.3 · p. 4Results
56.21 mm body and 51.59 mm hand global MPJPE were reported; reconstructions were more robust than MAMMA on partial-body, occluded, and truncated views.
Source: §4.2 · p. 5, §4.3 · p. 5Takeaways & Limitations
The dense motion resource may support embodied-AI motion learning and human-activity applications including skill assessment, coaching, and forecasting.
Source: §2 · p. 3
Abstract
from arXiv · showhide
Ego-Exo4D is a large-scale dataset providing synchronized egocentric and multi-view exocentric video, a rich resource for skill learning and assessment, procedural activity understanding, and embodied AI. However, the dataset ships with only sparse 3D human pose annotations, and reconstructing dense human motion from its multi-view captures is nontrivial. To this end, we present Ego-Exo4D-HM, a large-scale dataset of 4D human motion reconstructions for Ego-Exo4D's captures, and release the accompanying reconstruction pipeline. The code, dataset, and documentation can be found at https://abhiram824.github.io/egoexo4d_human_meshes.
1 Introduction
Ego-Exo4D’s synchronized egocentric and multi-view exocentric captures offer rich data, but its 3D annotations are sparse and reconstruction systems can be brittle at this scale. The paper releases a pipeline and SMPL-H motion sequences to make reconstructed motion directly usable.
1 Introduction
- Ego4D and Ego-Exo4D provide thousands of video hours across activities and participants, supporting potential uses in skill learning, procedural understanding, embodied AI, and coaching.The paper also positions the dataset for robot imitation learning and feedback on physical performance.Source: §1 · p. 1
- Ego-Exo4D combines synchronized egocentric and multiple exocentric cameras, but existing 3D annotations are sparse and reconstruction systems can be brittle at this scale.The multi-view setup should improve 3D human perception because exocentric cameras observe the performer’s full body.Source: §1 · p. 1
- The released pipeline recovers 4D human motion and provides SMPL-H sequences for Ego-Exo4D captures, so users can use the data without reprocessing raw video.Figure 1 shows recovered SMPL-H meshes overlaid on exocentric video across reconstructed sequences.Source: §1 · p. 1, Figure 1 · p. 2
2 Background
Prior work uses human motion and activity video for embodied AI and human-skill applications. This paper situates its multi-view reconstruction approach among video-based mesh recovery and egocentric pose methods.
Related reconstruction methods
- The approach builds on video-based mesh recovery, using optimization to combine pose and root-trajectory estimates with multi-view constraints, unlike feed-forward methods that find multiple views challenging.It is related to calibrated multi-view EasyMocap, while also handling egocentric captures and using more recent pose-estimation approaches.Source: §2 · p. 2
Egocentric and multi-view methods
- Prior egocentric pose methods use head-camera SLAM alone or add estimated hand observations; Ego-Exo4D’s exocentric cameras observe the body from multiple views.MAMMA performs markerless multi-view motion capture directly from raw video streams.Source: §2 · p. 3
Applications
- Human motion and activity data have supported humanoid controllers, visuomotor policies, and action-conditioned video or world models; related applications include skill feedback, interaction prediction, and motion editing.The paper anticipates that dense 3D motion estimates at scale could support these directions.Source: §2 · p. 3
3 Methods
The pipeline adapts SLAHMR to Ego-Exo4D’s calibrated exocentric cameras and egocentric SLAM, recovering world-frame motion through single-view keypoints, triangulation, and SMPL-H optimization.
3 Methods
- The person’s state comprises per-frame root orientation, joint pose, translation, and a time-invariant shape, which SMPL-H maps to mesh vertices and joints.The representation targets world-frame 3D body and hand motion from synchronized egocentric and exocentric video.Source: §3 · p. 3
3.1 Single-view pose estimation
- The single-view stage selects the wearer using the Aria SLAM-projected head position, then estimates 67 body and hand keypoints with ViTPose and HaMeR.A Mask R-CNN detector with a RegNetY-4GF backbone identifies candidate people in each exocentric view.Source: §3.1 · p. 4
3.2 Triangulation
- Each keypoint is triangulated independently per frame by minimizing reprojection error across calibrated views, requiring at least two views; RANSAC rejects hand and wrist outlier detections.Hand and wrist points receive this additional outlier filtering because they are more prone to occlusion and misdetection.Source: §3.2 · p. 4
3.3 SMPL-H Optimization
- SMPL-H optimization estimates per-frame translation, orientation, and pose plus one shape per take, freezing calibrated cameras to recover metric world-frame motion without a motion prior.The triangulated 3D evidence already constrains global motion.Source: §3.3 · p. 4
4 Results
The pipeline produces metric-world reconstructions and a curated dataset; reported evaluations include global keypoint errors and a qualitative comparison with MAMMA.
4.1 Curated Dataset
- 2,649 takes remain after 551 (17.1%) fail filters for reprojection error (>10% samples over 50 px) or triangulation coverage (<50%); reconstructed motion totals 104.59 hours.The pipeline was run on approximately 3,200 videos.Source: §4.1 · p. 4
4.2 Quantitative Results
- 56.21 mm body and 51.59 mm hand global MPJPE were reported; reconstructions were more robust than MAMMA on partial-body, occluded, and truncated views.The body result covers 845 annotated takes and the hand result 190 annotated takes.Source: §4.2 · p. 5, §4.3 · p. 5
4.3 Comparison with MAMMA
- The qualitative comparison reports more robust reconstructions than MAMMA for partial body views, occlusions, and truncations common in Ego-Exo4D captures.Figure 2 presents the pipeline-versus-MAMMA comparison.Source: §4.3 · p. 5, Figure 2 · p. 5