PaperlayerView exact revision on arXiv

Source-linked AI summary

Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild

Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, Rowan Postyeni, Luis Pesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, Kevin Bailey, David Soriano Fosas, C. Karen Liu, Ziwei Liu, Jakob Engel, Renzo De Nardi, Richard Newcombe

arXiv:2406.09905v2cs.CVcs.GR

TL;DR

Wearable devices provide insufficient self-observations, and existing datasets are limited in scale, modality, alignment, or contextual language descriptions. Nymeria combines full-body motion capture with synchronized and localized egocentric device data, third-person observation, and coarse-to-fine in-context language descriptions. The dataset contains 300 daily activities from 264 participants, and its language annotations comprise 310.5K sentences and 8.64M words.

  • Problem

    Wearable devices provide insufficient self-observations, and existing datasets are limited in scale, modality, alignment, or contextual language descriptions.

    Source: §1 · pp. 2–3

  • Method

    Nymeria combines full-body motion capture with synchronized and localized egocentric device data, third-person observation, and coarse-to-fine in-context language descriptions.

    Source: §1 · p. 3, Scenarios · p. 6

  • Results

    The dataset contains 300 daily activities from 264 participants, and its language annotations comprise 310.5K sentences and 8.64M words.

    Source: §3.4 · pp. 10–11

  • Takeaways & Limitations

    The dataset is described as useful for in-context character animation and as a resource for simulating 3D scenes and sensors, especially alongside motion priors.

    Source: §4.1 · p. 12

    The mocap suit and wristbands affect video appearance and restrict some motion ranges; the dataset covers only a portion of daily activities and omits common public scenarios.

    Source: §5 · p. 14

Abstract

from arXiv · show

We introduce Nymeria - a large-scale, diverse, richly annotated human motion dataset collected in the wild with multiple multimodal egocentric devices. The dataset comes with a) full-body ground-truth motion; b) multiple multimodal egocentric data from Project Aria devices with videos, eye tracking, IMUs and etc; and c) a third-person perspective by an additional observer. All devices are precisely synchronized and localized in on metric 3D world. We derive hierarchical protocol to add in-context language descriptions of human motion, from fine-grain motion narration, to simplified atomic action and high-level activity summarization. To the best of our knowledge, Nymeria dataset is the world's largest collection of human motion in the wild; first of its kind to provide synchronized and localized multi-device multimodal egocentric data; and the world's largest motion-language dataset. It provides 300 hours of daily activities from 264 participants across 50 locations, total travelling distance over 399Km. The language descriptions contain 301.5K sentences in 8.64M words from a vocabulary size of 6545. To demonstrate the potential of the dataset, we evaluate several SOTA algorithms for egocentric body tracking, motion synthesis, and action recognition. Data and code are open-sourced for research (c.f. https://www.projectaria.com/datasets/nymeria).

1 Introduction

Wearable body-motion understanding is constrained by limited and incomplete data, while current capture and annotation approaches also face challenges in long-duration recording and multimodal alignment. Nymeria addresses these gaps with a large, in-the-wild multimodal motion dataset and hierarchical language descriptions.

1 Introduction

  • Existing real datasets have limited scale, diversity, or modality, while simulations lack realism and completeness.The introduction identifies data availability as a critical limiting factor for research.Source: §1 · p. 2
  • Vision-based capture is constrained by line of sight and a limited capture volume, while inertial methods suffer from dead-reckoning and weaker global positioning accuracy.The cited approaches include optical-marker or camera systems and inertial-based tracking.Source: §1 · p. 2
  • Existing synchronization approaches can be unreliable or intrusive over long recordings, and prior language annotations often lack scene context or corpus scale.The introduction also notes that existing datasets mostly record short motions.Source: §1 · pp. 2–3
  • Nymeria provides 300 hours of in-the-wild activity from 264 participants across 20 scenarios and 50 locations, with synchronized multimodal egocentric devices and motion-language annotations.The corpus contains 310.5K sentences, 8.64M words, and a vocabulary of 6,545 words.Source: §1 · p. 3

2 Related Works

Prior motion datasets vary in scale, capture setting, and available modalities, while motion-language resources often provide sparse or brief descriptions. Nymeria is presented against these gaps as a large-scale dataset with multimodal capture and hierarchical narration.

Motion datasets – scale, multimodal, in-the-wild and perspectives

  • AMASS unifies marker-based datasets into SMPL but lacks scene context; other motion datasets use monocular or multi-view cameras, suits, or simulation, each with different capture constraints.Vision methods require line of sight, while simulations can have a domain gap because they do not reproduce real sensor noise characteristics.Source: §2 · pp. 3–4
  • Egocentric motion datasets remain sparse, with prior work limited in ground-truth, parametric body motion, scale, diversity, or modality.Nymeria is designed to address these gaps; Table 1’s caption reports its dataset-scale statistics without establishing a comparison to other datasets.Source: Table 1 · p. 3, §2 · p. 4

Motion with natural language

  • Motion-language resources range from action labels and brief descriptions to fine-grained narration, while Ego4D and EgoExo4D provide atomic-action descriptions without ground-truth motion or parametric representations.Nymeria provides in-context coarse-to-fine narration by annotators, at a scale described as a magnitude larger than prior works.Source: §2 · p. 5

Egocentric body tracking, synthesis and action recognition

  • Egocentric tracking research uses wearable sensors and synthesis to address under-observed body motion, while Nymeria is constructed as an offline dataset with dense IMUs and drift correction for ground truth.The related-work discussion also notes growing interest in text-driven motion synthesis and language-linked motion understanding.Source: §2 · p. 5

3 Building Nymeria Dataset

Nymeria combines wearable and motion-capture recordings with synchronized views, global localization, and hierarchical language annotations. Its statistics describe a broad collection of participants, locations, activities, and language data.

3.1 Data collection setup

  • Participants wear an XSens suit, Project Aria glasses, and miniAria wristbands, while an observer records a third-person view.The suit has 17 inertial trackers; the glasses capture RGB, grayscale, eye-tracking, IMU, audio, and other sensor data, while the wristbands omit microphones, barometer, magnetometer, and eye-tracking cameras.Source: §3.1 · p. 6, Recording protocols · p. 6
  • Twenty indoor and outdoor scenarios use high-level activity guidelines and operator prompts to encourage natural interactions and varied motion.Scenarios include cooking, working, hiking, biking, dining, and sports.Source: Scenarios · p. 6

3.2 Data processing

  • MPS globally aligns recordings into one metric 3D world, then XSens body poses are registered by comparing many local-velocity clips because XSens dead-reckoning accumulates drift, using 4.2 ms segments.The model has 79 anatomical landmarks used to retarget motion onto an anatomically inspired human model; Aria’s movement during recording is a main source of inaccuracy.Source: Full-body mocap and retargetting · pp. 7–8, 6DoF localization and mapping with… · p. 9

3.3 In-context motion-language description

  • Annotators use synchronized egocentric, third-person, and motion-rendered views to produce fine-grained motion narration, atomic actions, and 30-second activity summaries.The narration and atomic-action tasks use clips shorter than 5 seconds; narration describes posture, direction, velocity, interactions, and attention, while atomic actions favor verbs.Source: §3.3 · pp. 9–10

3.4 Statistics

  • The dataset contains 300 daily activities from 264 participants, totaling 1,200 sequences averaging 15 minutes, with accumulated trajectories of 399 km for headsets and 1,053 km for both wristbands.Source: §3.4 · p. 10
  • The dataset contains 310.5K sentences, 8.64M words, and a vocabulary of 6,545 distinctive words across three annotation tasks.Motion narration covers 38.6 hours, atomic action 207 hours, and activity summarization 196 hours; their average clip lengths are 5, 5, and 30 seconds, respectively.Source: §3.4 · p. 11

4 Benchmark Tasks and Baselines

The section outlines research opportunities in motion, spatial and video understanding, and simulation, then evaluates Nymeria through tracking and synthesis, motion tokenization, and motion-to-text baselines.

4.1 Research opportunities

  • Nymeria supports motion tasks including full-body tracking, synthesis, forecasting, path planning, action recognition, and behavior analysis, with multimodal settings such as gaze-conditioned prediction.The section also suggests action recognition from headset and wristbands and interaction generation from language.Source: §4.1 · p. 11
  • Localized egocentric videos, synchronized with motion and contextual narrations, support spatial reasoning, video understanding, image retrieval, and relocalization.Multiple videos per location are aligned in global coordinates.Source: §4.1 · pp. 11–12

4.2 Motion tasks baselines

4.1 Research opportunities

  • Nymeria is proposed for in-context character animation, 3D-scene simulation, and sensor simulation, particularly for IMUs, magnetometers, and barometers with motion priors.The authors note that synthetic data are typically seeded in the real world.Source: §4.1 · p. 12

4.2 Motion tasks baselines

  • Nymeria motion data achieve similar VQ-VAE performance to AMASS, with reconstruction further improved by product quantization, larger codebooks, and smaller latent dimensions.The ablation evaluates product quantization, codebook size, and latent dimensions.Source: Motion tracking and synthesis from… · p. 13, Table 5 · p. 13
  • MotionGPT and TM2T are evaluated for motion-to-text on 30 hours of narrations; results are worse than HumanML3D and KIT, whose descriptions are less complex and diverse.TM2T performs worse than MotionGPT, which uses a T5 language prior; the authors expect full narration data to improve performance.Source: Motion and language · p. 14, Table 6 · p. 14

5 Conclusions and Discussions

Nymeria combines large-scale motion, multimodal egocentric data, and hierarchical language annotations, while the authors identify wearable-related constraints and gaps in activity coverage.

5 Conclusions and Discussions

  • Nymeria contains 260M body poses from 264 participants across 50 locations, with synchronized 6DoF tracking, 3D scene points, gaze, 201.2M egocentric images, 11.7B IMU samples, and 10.8M gaze points.The conclusion reports 399 km of participant travel and 310.5K sentences in 8.64M words, spanning fine-grained narration, atomic actions, and activity summaries.Source: §5 · p. 14
  • The mocap suit and wristbands cause unnatural video appearance and restrict some motion, XSens quality is affected by calibration and body measurements, and common public scenarios remain uncovered.The dataset covers only a portion of daily activities.Source: §5 · p. 14
  • The authors cite consent, de-identification, minimum data retention, and a permissive research license as privacy measures for a dataset involving personal data.Source: §5 · p. 14

Appendix

  • The appendix covers capture setup, data processing, recording scenarios and locations, and baseline-experiment details; the hardware profiles specify video, eye-tracking, IMU, audio, and mocap recording rates.GNSS, WiFi, and Bluetooth are disabled on Project Aria glasses for privacy; miniAria lacks audio, magnetometer, and barometer sensors.Source: Appendix · p. 16, Recording profile · p. 16

B.1 Project Aria MPS

Project Aria’s machine perception service provides calibrated sensor streams, localization, gaze estimates, and point-cloud maps, with raw outputs included in the dataset.

B.1 Project Aria MPS

  • Figure 9 compares grayscale and RGB views from Aria glasses with views from miniAria wristbands across four common body postures.Source: Figure 9 · p. 17
  • All Aria and miniAria sensors have intrinsic and extrinsic calibration, with online calibration correcting small temperature- or stress-related frame deformations.Camera calibration uses a spherical base model with radial, tangential, and thin-prism distortion coefficients.Source: §B.1 · p. 18
  • Visual-inertial odometry and SLAM place recordings in a common metric, gravity-aligned frame, providing millimeter-accurate 6DoF poses and 1 kHz motion between camera frames.Both closed-loop aligned and open-loop dead-reckoned trajectories are provided.Source: §B.1 · p. 18
  • Eye gaze is represented by outward-facing rays from each eye, estimating look direction and approximately the metric distance of focus.Optional phone-based gaze calibration personalizes the estimated direction.Source: §B.1 · p. 18
  • Moving Aria devices triangulate temporary static scene points from consecutive stereo monochrome frames and provide raw 2D observations for tracking when points are seen.The observation times help account for furniture moved within scripted scenes.Source: §B.1 · p. 18

B.2 Motion retargeting

The retargeting pipeline maps XSens landmarks to a parameterized anatomical body model, using nonlinear least squares with pose, collision, and optional smoothness constraints.

B.2 Motion retargeting

  • The model has 159 skeletal joints, but retargeting uses 89 body joints to represent ground-truth motion while leaving face and finger joints unchanged.The 159-joint model deforms a quadrangle mesh through linear blend skinning.Source: §B.2 · pp. 18–19
  • Of the 89 body joints, one represents global motion, 35 follow human bones, and 53 helper joints distribute twist to reduce linear-blend-skinning artifacts.The helper joints are driven linearly by the 35 bones.Source: §B.2 · p. 19
  • The model uses 12 identity parameters and 58 pose parameters, with reduced joint rotations encoding limits and synergies; the mesh remains template-shaped because XSens supplies no body shape.Pose parameters comprise 6DoF global transformation and 52 Euler angles; elbows and knees each have one rotational degree of freedom.Source: §B.2 · pp. 19–20
  • For each motion, XSens landmark locations are inputs to a Levenberg–Marquardt inverse-kinematics optimization that estimates user identity, landmark offsets, and per-frame poses.Manually defined landmark locations serve only as an initial guess, with local offsets solved during optimization.Source: §B.2 · p. 20
  • Quadratic parameter-limit penalties and tapered-capsule collision penalties address XSens artifacts; an optional smoothness term penalizes pose changes between consecutive frames.Source: §B.2 · p. 20
  • To avoid solving a long sequence jointly, the method estimates identity parameters and landmark offsets from a uniformly sampled subset, then solves poses frame by frame.The example subset contains 200 frames.Source: §B.2 · p. 20

B.3 In-context motion narration

Nymeria’s in-context narration protocol aligns descriptions with synchronized egocentric and third-person video and rendered full-body motion. It uses three annotation levels, from detailed motion narration to broader activity summaries.

Annotation goal

  • The 25 annotators write English descriptions intended to connect full-body motion, egocentric perception, and language, including body poses and interactions.The protocol aims for in-context descriptions rather than motion descriptions detached from their environment.Source: §B.3 · p. 20

Tool interface and guidelines

Motion narration

  • Motion narration uses 3–5-second segments and asks about full-body motion, hand and arm movement, leg and foot movement, and focus of attention.The questions prompt descriptions of motion direction, intent, and environment as well as detailed body movement.Source: Guidelines and narration questions · p. 22

Atomic actions

  • Atomic-action annotations use 3–5-second segments and emphasize verbs to describe actions, posture, movement direction, and interactions.Annotators answer one question per segment, focusing on actions rather than detailed body poses.Source: Guidelines and narration questions · p. 22

Activity summarization

  • Activity summarization uses 15–30-second segments to summarize the major activity and its interactions with people, objects, and the scene.This is the coarsest of the three annotation levels.Source: Guidelines and narration questions · p. 22

C.1 Recording scenarios

The recording protocol covers 20 scripted indoor and outdoor scenarios, using everyday activities and interactions to capture varied motion. Scenario statistics report recording duration and head and wrist trajectory lengths, while distinguishing clean aligned data from longer recorded totals.

Home scenarios

  • The home scenarios include relaxing, searching for misplaced objects, and giving visitors a house tour, with activities such as walking, looking, greeting, and conversation.Searching recordings involve finding 5–10 hidden objects during a 15-minute session.Source: Definitions · pp. 23–24

Exercise and dance

  • Stretching, cardio workouts, and dancing use video instructions, with dancing designed to capture frequent arm swings and body rotation.Participants can use workout equipment when available, and their comfort level is checked before exercise recordings.Source: Definitions · p. 24

Domestic activities

  • Cooking, shared meals, creating a mess, and housekeeping capture domestic tasks such as preparing food, eating with others, misplacing objects, and cleaning.Housekeeping is usually recorded with a different participant cleaning up the mess created earlier.Source: Definitions · p. 24

Routines and social play

  • Laundry and packing, game night, and charades cover household routines and social play, including games and exaggerated body-language performances.Charades prompts participants to act out themes for others to guess.Source: Definitions · pp. 24–25

By the desk

  • The desk-work scenario records computer tasks, writing, doodling, and crafting in a work-from-home setting.Source: Definitions · p. 25

Command-following scenarios

  • Command-following scenarios include predefined motions and mini real-life events, capturing locomotion and long-tailed activities such as making tea or watering plants.Simon Says uses operator instructions and audio recordings as pseudo-ground-truth action labels.Source: Definitions · p. 25

Work and outdoor scenarios

  • Office work, hiking, outdoor recreation, and party preparation extend recordings to onsite work, trails, sports, and collaborative decorating.Party scenarios are mostly recorded with two participants.Source: Definitions · pp. 25–26

Recording statistics

  • 300 hours are the clean data with all modalities aligned and high-quality motion; scenario-table totals are longer because they include eye calibration and additional poor-quality-motion data.The scenario table breaks down recording hours and trajectory lengths by head, left wrist, and right wrist.Source: Table 7 · p. 23, Definitions · p. 26

C.2 Locations

Nymeria spans 50 locations, including houses, a cafeteria, an office building, and connected areas of a campus. The location summary reports recording duration and head and wrist trajectory lengths, alongside information about each site's layout.

Location coverage

  • 50 locations comprise 47 houses, a cafeteria with an outdoor patio, a multistory office building, and campus areas with parking and hiking or biking trails.The three campus areas are aligned into shared world coordinates using additional basemap recordings and are merged as location BX.Source: §C.2 · p. 26

Location statistics

  • Location statistics report days onsite, recorded hours, head and wrist trajectory lengths, and site layout; the table identifies houses with an AB prefix and the merged campus as BX.Its layout fields include bedrooms, bathrooms, multistory status, and listed rooms or spaces.Source: §C.2 · p. 26, Table 8 · p. 27

D Benchmarks

The benchmark section describes baseline methods and qualitative visualizations for motion tracking, reconstruction, and understanding. It adapts existing approaches to Nymeria’s Xsens skeleton and data, while preserving their core architectures where possible.

Qualitative visualizations

Egocentric body tracking and synthesis

  • AvatarPoser predicts full-body pose from head and hand positions over the preceding 40 frames, while BoDiffusion generates full-body motion from sequences of three-point inputs.Both are adapted to the Xsens skeleton and retrained on Nymeria data; BoDiffusion takes about three times longer to converge than on AMASS.Source: AvatarPoser · p. 29, BoDifffusion · p. 29

Egocentric body tracking

  • EgoEgo’s benchmark uses supplied head positions as input to its diffusion-based full-body prediction stage, omitting its visual localization stage.The original method estimates head trajectories from camera images before generating full-body motion.Source: EgoEgo · p. 29

Motion representation

  • The VQ-VAE encodes motion into discrete tokens, using a convolutional encoder and decoder designed to process variable-length sequences.Its training uses 1-second windows, and quantization techniques are applied to improve codebook usage and reconstruction performance.Source: VQ-VAE · pp. 29–30

Motion understanding

  • TM2T and MotionGPT are evaluated for motion understanding by mapping motion tokens to language; MotionGPT is tested after motion-to-text and text-to-motion training.Both use the described motion VQ-VAE as their motion tokenizer.Source: TM2T · p. 30, MotionGPT · p. 30

Related papers

Loading 2406.09905v2…