Source-linked AI summary
Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset
Claire McLean, Makenzie Meendering, Tristan Swartz, Orri Gabbay, Alexandra Olsen, Rachel Jacobs, Nicholas Rosen, Philippe de Bree, Tony Garcia, Gadsden Merrill, Jake Sandakly, Julia Buffalini, Neham Jain, Steven Krenn, Moneish Kumar, Dejan Markovic, Evonne Ng, Fabian Prada, Andrew Saba, Siwei Zhang, Vasu Agrawal, Tim Godisart, Alexander Richard, Michael Zollhoefer
TL;DR
Motion datasets often trade scale against quality and completeness, while also lacking breadth across motion domains or modalities. Embody 3D collects multimodal 3D motion across single- and multi-person activities. It reports 500 individual hours from 439 participants and more than 54 million tracked 3D frames.
Problem
Existing motion datasets trade scale against tracking quality and completeness, and often specialize in limited motion domains or omit audio and text.
Source: §1 · pp. 1–2Method
Embody 3D collects multi-view 3D motion across varied single- and multi-person activities, with body and hand tracking, body shape, audio, and text annotations.
Source: §1 · p. 2Results
The dataset contains 500 individual hours from 439 participants and more than 54 million tracked 3D motion frames.
Source: §1 · p. 2Takeaways & Limitations
By combining tracking, modalities, and motion domains in one dataset, Embody 3D addresses coverage gaps described for existing resources.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
The Codec Avatars Lab at Meta introduces Embody 3D, a multimodal dataset of 500 individual hours of 3D motion data from 439 participants collected in a multi-camera collection stage, amounting to over 54 million frames of tracked 3D motion. The dataset features a wide range of single-person motion data, including prompted motions, hand gestures, and locomotion; as well as multi-person behavioral and conversational data like discussions, conversations in different emotional states, collaborative activities, and co-living scenarios in an apartment-like space. We provide tracked human motion including hand tracking and body shape, text annotations, and a separate audio track for each participant.
1 Introduction
Existing motion datasets trade off scale, tracking quality, completeness, and breadth of motion or modalities. Embody 3D presents one large-scale 3D dataset spanning varied behaviors with comprehensive tracking and additional modalities.
1 Introduction
- Existing datasets trade scale against quality and completeness, while many 3D collections omit hands or body shapes and specialize in narrow motion domains or lack audio and text.Table 1 contrasts scalable 2D datasets with high-quality but typically smaller 3D datasets, and presents Embody 3D as covering these gaps.Source: §1 · pp. 1–2, Table 1 · p. 2
- 2D video can scale to thousands of hours, but monocular depth ambiguity, occlusion, blur, and inconsistent world space limit its precision and spatial consistency.These constraints can be especially consequential around hands and when multiple actors appear.Source: §1 · pp. 1–2
- Embody 3D provides 500 individual hours from 439 participants and over 54 million 3D motion frames, combining full-body, hand, and body-shape tracking with audio and text.Its multi-view collection uses 80 cameras and spans single-person motions and multi-person conversations, collaborative activities, and co-living scenarios.Source: §1 · p. 2, Table 1 · p. 2
2 Dataset Details
The dataset spans seven subcategories, from prompted and hand-focused single-person motion to multi-person conversations, activities, and apartment-like co-living scenarios. The categories vary in duration, participants, and available assets.
Charades
- Charades provides 88.9 hours of prompted motions, with participants performing 15-second prompts and tracked motion paired with the prompt text.The subcategory includes 221 participants.Source: §2 · p. 3
Hand Interactions
- Hand Interactions contains 111.3 hours of hand-focused segments, including self-contact between the hands or between hands and body.The subcategory includes 137 participants.Source: §2 · p. 3
Locomotion
- Locomotion provides 21 hours of instructed walking, running, and jumping styles from 46 participants.Source: §2 · p. 3
Dyadic Conversations
- Dyadic Conversations contributes 59.4 individual hours from 86 participants, spanning prompted topics, varied emotional states, and unguided conversation.Source: §2 · p. 3
Multi-Person Conversations
- Multi-Person Conversations offers 125.2 individual hours from 210 participants, extending dyadic conversations to groups and allowing use of chairs, a high-table, or a couch.Source: §2 · p. 3
Scenarios
- Scenarios provides 49.2 individual hours from 77 participants performing collaborative or competitive activities, including games and furniture assembly.The subcategory includes object and furniture interactions.Source: §2 · p. 3
Day in the Life
- Day in the Life contains 46.4 individual hours from 77 participants interacting in apartment-like settings around co-living, hosting, or group activities.The paper identifies it as the dataset’s most challenging motion subcategory.Source: §2 · p. 3
Dataset overview
- Table 2 summarizes available assets across the seven subcategories, and a checkmark can indicate only high-level labels rather than fine-grained text annotations.Source: §2 · pp. 2–3, Table 2 · p. 3
3 Collection System and Dataset Acquisition
Embody 3D collects single-person motions and interactive scenarios in a multimodal camera-and-audio system. The acquisition combines supervised sessions with detailed annotations for selected categories.
3 Collection System and Dataset Acquisition
- The collection system uses 80 high-resolution cameras and five 128-element microphone arrays to capture video and audio.The camera array records at 30 fps; the microphone arrays record 10th-order ambisonics.Source: §3 · pp. 3–4
- Single-person categories are recorded with multiple participants simultaneously, while participants in interactive scenes engage with one another.The system includes 80 cameras, and the microphone arrays provide 640 audio channels.Source: §3 · pp. 3–4, Figure 1 · p. 4
- Research assistants brief and guide participants, monitor recordings in real time, and flag quality issues.Participants complete body-shape calibration and provide informed consent before collection.Source: §3 · p. 4
- Human-generated annotations for Scenarios and Day-in-the-Life segments describe scenes, individual motion, and participants’ emotional states.Emotional-state labels draw on facial expression, pose, and speech.Source: §3 · p. 4
4 Data Processing
The processing pipeline synchronizes and calibrates capture data, estimates participant shapes and poses, and separates speech by participant. Human review filters out segments with low tracking-quality scores.
4 Data Processing
- The pipeline converts raw recordings into multi-person poses and audio, including tracked 3D motion, body shapes, and speech-separated channels.Figure 2 gives a schematic overview of the processing stages.Source: §4 · p. 4
4.1 Synchronization and Calibration
- Camera timestamps are clustered into a global frame list, outlier cameras and sparse frames are filtered, and audio samples are aligned to synchronized segments.Geometric calibration establishes a shared floor-centric world coordinate system; average p50 reprojection error is under 0.2 px.Source: §4.1 · p. 5
4.2 Participant Shape Estimation
- Four calibration poses are used to estimate each participant’s mean body shape by optimizing coefficients of a linear 3D human shape model.The poses are A, T, C, and T-Rex.Source: §4.2 · p. 5, Figure 3 · p. 5
4.3 Multi-Person Pose Estimation
- Per-camera person detections yield 308 face and body keypoints, which are grouped across views and frames before participant identities are assigned using face embeddings.Identity matching compares sampled face crops with reference-face embeddings using cosine similarity and the Hungarian algorithm.Source: §4.3 · pp. 5–6, Figure 4 · p. 6
- 3D keypoints are triangulated with RANSAC and refined using point-to-ray distance, temporal smoothness, and bone-length constraints before pose tracking estimates joint rotations.A pretrained pose prior regularizes the joint rotations.Source: §4.3 · p. 6
4.4 Beamforming
- A beamforming algorithm separates speech for each participant from 640 microphone channels, with hyperparameters tuned to balance separation and distortion.The release also includes non-separated audio from a central reference microphone.Source: §4.4 · p. 6
4.5 Quality Assurance
- Segments scoring below 2.5 on the one-to-five tracking-quality scale are discarded; retained segments were observed to be free of significant errors.Reviewers overlaid tracked motion on four camera views; penalties were mostly for minor misalignments in subtle details.Source: §4.5 · p. 6