PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video

Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, Jian Zhang

arXiv:2505.11709v3cs.CVcs.LGcs.RO

TL;DR

Dexterous-manipulation imitation learning lacks Internet-scale data, while existing egocentric videos generally lack manipulation focus and native 3D hand annotations. EgoDex addresses this gap with a large egocentric dataset paired with 3D pose tracking and benchmarks hand-trajectory policies, whose performance depends on sampling, prediction horizon, and visual goal conditioning.

  • Problem

    Dexterous manipulation lacks an Internet-scale corpus, and existing large egocentric datasets generally lack manipulation focus and paired 3D dexterous annotations.

  • Method

    EgoDex pairs large-scale egocentric video with detailed 3D hand pose annotations and trains and evaluates imitation-learning policies for hand trajectory prediction.

  • Results

    EgoDex contains 829 hours and 338000 demonstrations across 194 tasks; flow matching outperforms other models by up to 34% for K = 5 and K = 10, while visual goal-conditioning reduces average distance by 22% and final distance by 53%.

  • Takeaways & Limitations

    EgoDex provides a large, diverse benchmark for studying dexterous manipulation from egocentric video and measuring hand-trajectory prediction progress.

  • Takeaways & Limitations

    EgoDex has limited background and scene diversity, and its dexterous annotations can be imperfect during heavy occlusion or very high-speed motions.

Abstract

from arXiv · show

Imitation learning for manipulation has a well-known data scarcity problem. Unlike natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. One appealing option is egocentric human video, a passively scalable data source. However, existing large-scale datasets such as Ego4D do not have native hand pose annotations and do not focus on object manipulation. To this end, we use Apple Vision Pro to collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date. EgoDex has 829 hours of egocentric video with paired 3D hand and finger tracking data collected at the time of recording, where multiple calibrated cameras and on-device SLAM can be used to precisely track the pose of every joint of each hand. The dataset covers a wide range of diverse manipulation behaviors with everyday household objects in 194 different tabletop tasks ranging from tying shoelaces to folding laundry. Furthermore, we train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area. By releasing this large-scale dataset, we hope to push the frontier of robotics, computer vision, and foundation models. EgoDex is publicly available for download at https://github.com/apple/ml-egodex.

1 INTRODUCTION

EgoDex addresses the scarcity and scaling challenges of dexterous-manipulation data by pairing passively scalable egocentric video with detailed 3D hand tracking. It provides a large, diverse manipulation dataset and benchmarks imitation-learning policies for hand trajectory prediction.

  • Egocentric video with paired 3D hand pose annotations offers a passively scalable alternative to teleoperation for collecting manipulation data.The approach is motivated by the difficulty of scaling deliberate physical-robot data collection.
  • EgoDex contains 829 hours of 30 FPS video, 90 million frames, and 338000 demonstrations across 194 tabletop manipulation tasks.
  • EgoDex uses calibrated cameras and on-device SLAM to capture detailed 3D pose information for the head, arms, wrists, and every finger joint.The 30 FPS 1080p video has a wide field of view and is designed to capture much of the human manipulation context.
  • The dataset emphasizes diverse active manipulation beyond pick-and-place, including unscrewing bottle caps, flipping pages, and plugging chargers into sockets.Its tasks use everyday household objects and are performed on tabletops.
  • The paper systematically evaluates imitation-learning policies for hand trajectory prediction to assess the state of the art and identify future challenges.The authors also aim for broader utility in computer vision, video generation, and world modeling.

2 RELATED WORK

Prior manipulation datasets trade off scale, scalability, hardware specificity, annotation richness, or behavioral diversity. EgoDex combines large-scale egocentric video with native dexterous annotations and broader manipulation coverage, while addressing limitations of postprocessed Internet video and active collection systems.

  • Large-scale manipulation datasets: EgoDex has the largest amount of trajectories, tasks, and frames by a large margin among the compared datasets, with language, camera-extrinsic, and dexterous annotations.Dexterous annotation denotes multi-finger hand poses rather than lower-fidelity wrist-only or parallel-jaw-gripper data.
  • Large-scale manipulation datasets: Teleoperation datasets require labor-intensive physical-robot operation and may not generalize beyond their collected hardware embodiments and camera viewpoints.
  • Learning from human video: Ego4D and EPIC-KITCHENS are scalable and hardware-agnostic but typically do not focus on manipulation or provide paired 3D dexterous annotations.
  • Learning from human video: Hand-object-interaction datasets often provide 3D hand poses but are orders of magnitude smaller and emphasize grasping over diverse, long-horizon manipulation.
  • Scalable methods for robot data collection: Alternative robot-free collection systems remain actively collected, requiring demonstrators to intentionally provide data rather than producing it as a passive byproduct.
  • Learning from human video: Postprocessing raw Internet video with hand-prediction networks can suffer without multiple viewpoints and known camera extrinsics, whereas EgoDex records these during collection.
  • Learning from human video: EgoMimic is the closest prior work, but EgoDex scales from around 4 hours to 829 hours and uses a broader task distribution with more dexterous annotations.

3 EGODEX DATASET

EgoDex is a large-scale egocentric manipulation dataset combining high-resolution video, structured pose annotations, and diverse tabletop behaviors. Its collection design supports scale through reset-efficient tasks, while its scope emphasizes behavioral rather than scene diversity.

  • Dataset scale: 829 hours of 1080p, 30 Hz egocentric video comprise 338,000 episodes across 194 tabletop tasks and 90 million frames.The dataset also includes natural-language, camera, and hand-pose annotations and occupies 2.0 TB.
  • Data collection: Its Vision Pro collection setup provides unobstructed egocentric views and pose tracking without additional hardware apparatus.ARKit records demonstrations with bare hands, while high-frequency passthrough and wide field of view capture the collector’s visual perspective.
  • Annotations: EgoDex records RGB video, camera intrinsics and extrinsics, upper-body and hand-joint poses, confidence values, and natural-language manipulation annotations.The skeletal data covers 25 joints for each hand at 30 Hz.
  • Collection efficiency: Reversible and reset-free tasks increase data-collection yield by avoiding costly resets that are excluded from recorded data.Examples include inverse task pairs and actions whose final states naturally return to their initial-state distribution.
  • Behavioral diversity: EgoDex emphasizes dexterous behaviors such as tightening screws, tying shoelaces, dealing cards, flipping pages, and slotting batteries beyond basic pick-and-place.Most verbs have more than 10^3 demonstrations, contrasting with the narrower distribution reported for DROID.
  • Scope: Scene diversity is limited to tabletop environments because the work focuses on behavioral diversity rather than the Cartesian product of scenes and behaviors.The authors identify visual data augmentation as a way to introduce additional scene diversity.

4 EGODEX BENCHMARKS

EgoDex benchmarks hand-trajectory prediction from egocentric observations, skeletal poses, language, and optionally a visual goal. They use fixed horizons and reproducible held-out evaluation while accounting for multimodal human motion with best-of-K scoring.

  • Action representation: The action representation has 48 dimensions: both wrists’ 3D positions and 6D orientations plus five 3D fingertip positions per hand.Actions are predicted in fixed-horizon chunks as relative trajectories expressed in the current camera frame.
  • Trajectory prediction: Dexterous trajectory prediction maps egocentric images, skeletal poses, and a natural-language task description to future hand trajectories.The predicted output is an action chunk covering the selected horizon.
  • Inverse dynamics: Inverse dynamics additionally conditions hand-trajectory prediction on a goal image at the end of the horizon.This benchmark is interpreted as a visually goal-conditioned policy for handling multimodal natural motion.
  • Prediction horizons: Prediction horizons range from short settings such as H = 30 (1 second) to longer settings such as H = 90 (3 seconds).Longer horizons make the trajectory prediction task more difficult.
  • Evaluation protocol: The benchmarks reserve 1% of EgoDex as a fixed held-out test set, leaving 99% for flexible training and validation splits.This design makes evaluation reproducible without varying physical robot hardware or environments.
  • Evaluation metric: Best-of-K scoring samples K trajectories and reports the closest ground-truth match using mean Euclidean 3D keypoint error over time and 12 wrist-fingertip keypoints.The resulting value is interpretable as average positional error in meters and can improve with K for stochastic models.

5 EXPERIMENTS

The experiments evaluate 14 imitation-learning models and study architecture, policy representation, prediction horizon, visual goal-conditioning, model size, and dataset size. Encoder-decoder models generally outperform decoder-only models, goal-conditioning helps, and performance improves with more data but degrades over longer horizons.

  • Experimental setup: 14 models spanning two Transformer architectures and three policy representations are evaluated across horizon, goal-conditioning, dataset-size, and model-size experiments.All models are trained for 50,000 gradient steps with batch size 2048.
  • Architecture: Encoder-decoder models consistently outperform decoder-only counterparts by a small margin.This pattern holds across the evaluated models in Table 2.
  • Policy representations: Up to 34%, encoder-decoder flow matching outperforms other models for K = 5 and K = 10, while behavior cloning is about 15% better at K = 1.Diffusion and flow matching improve with larger K, whereas deterministic behavior cloning does not.
  • Prediction horizon: 31% and 21%, reducing the horizon from 2 seconds to 1 second improves average and final distance, while increasing it to 3 seconds worsens them by 18% and 11%.The longer horizon requires predicting 48-dimensional dexterous actions farther into the future.
  • Visual goal-conditioning: 22% and 53%, visual goal-conditioning reduces average and final distance, respectively.The goal image provides an endpoint anchor for the predicted trajectory and yields an inverse-dynamics baseline.
  • Scale: 0.045 average distance and 0.062 final distance are achieved by both 200-million- and 500-million-parameter models, while performance improves with larger datasets.The dataset-size result is reported for both average and final distance metrics.

6 RESEARCH USE CASES

EgoDex is positioned for uses beyond robot manipulation, including robotics, perception, video generation, and world modeling. Its egocentric video and dexterous hand annotations support studying actions, objects, contacts, and state changes.

  • Robotics: Robot applications include co-training with small robot datasets or pretraining on large-scale human data before supervised fine-tuning on robot data.These options are proposed for addressing the embodiment gap between humans and current robots.
  • Perception: EgoDex supports action recognition and human-object interaction detection, including identifying active objects, contact points, and object state changes.The passage connects these uses to established egocentric-video research on daily actions.
  • Video generation and world models: EgoDex may also support language-conditioned video generation and world-model research using its manipulation-centered egocentric observations.The passage frames these applications alongside recent diffusion-based video-generation advances.

7 CONCLUSION

The paper introduces EgoDex as a large-scale egocentric manipulation dataset with 3D pose annotations and evaluates hand-trajectory prediction policies on it. Its main scope limitation is restricted background and scene diversity, with annotations potentially imperfect under occlusion or fast motion.

  • Contributions: EgoDex combines egocentric video, 3D pose annotations, diverse dexterous manipulation tasks, and imitation-learning evaluation for hand trajectory prediction.The conclusion presents the dataset and policy evaluation as the paper’s central contributions.
  • Limitations: EgoDex has limited background and scene diversity, and its model-predicted dexterous annotations can be imperfect during heavy occlusion or very high-speed motions.Future work includes procedural background randomization and collection in more diverse environments.

A.1 ADDITIONAL EXPERIMENTS

Additional experiments examine out-of-distribution generalization and reproduce the prediction-horizon study with encoder-decoder flow matching. Similar tasks can match in-distribution performance, whereas more distant tasks perform worse, and longer horizons remain harder.

  • Out-of-distribution tasks: Some of six out-of-distribution tasks achieve performance comparable to in-distribution tasks, while more distant tasks perform worse.The results suggest generalization to OOD tasks that are at least somewhat similar to training tasks.
  • Prediction horizon: Encoder-decoder flow matching shows the same degradation in performance as prediction horizon increases.This confirms the horizon trend observed with the decoder-only behavior-cloning model.

A.2 COMPLETE LIST OF TASKS

EgoDex organizes 194 tabletop manipulation tasks into reset, reset-free, and reversible categories, including diverse household behaviors and a pick-and-place subset with language annotation.

  • Task organization: 194 tasks are organized as reset, reset-free, or reversible tasks, with each reversible task counted as a pair.The dataset lists 14 reset tasks, 28 reset-free tasks, and 76 reversible task pairs.
  • Deployment-oriented data: The basic pick place task provides diverse pick-and-place data together with high-quality language annotation for users interested in robot deployment.The passage specifically highlights this task as especially useful for deployment-oriented applications.
  • Reversible tasks: The reversible category includes dexterous behaviors such as braiding, shoelace tying, plugging sockets, and unscrewing bottle caps.These examples span reversible manipulation of garments, furniture, electronics, and household objects.
  • Reset-free tasks: Reset-free tasks include behaviors such as washing dishes, flipping pages, typing on a keyboard, and playing piano.These tasks are listed separately from the reset and reversible categories.

A.3 COMPLETE LIST OF SKELETAL JOINTS

EgoDex provides upper-body and hand skeletal annotations as SE(3) poses, with confidence values that must be interpreted differently for wrists and finger joints.

  • Pose representation: SE(3) poses are represented as 4 × 4 homogeneous transformation matrices for the annotated skeletal joints.The annotations cover the listed upper-body, hand, and finger joints.
  • Annotated joints: The skeletal list includes the hip, spine, neck, shoulders, arms, forearms, and both hands, alongside detailed finger joints.The listed joints include multiple body segments and named joints for each finger.
  • Joint naming: leftHand and rightHand refer to the wrists rather than generic whole-hand joints.This naming convention is important when interpreting the skeletal annotations and confidence values.
  • Model experiments: The experiments evaluate 14 models spanning architecture, optimization method, dataset size, prediction horizon, model size, and visual goal-conditioning.The models use current image observations and proprioceptive state without history; adding history may improve performance.
Loading 2505.11709v3…