HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos

Motivation

• Existing hand-object interaction methods lack the accuracy and speed needed for applications such as skill transfer, robotic assistance, contextual understanding, and new AR/VR inputs.The dataset is intended to accelerate research toward these applications.Source: §1 · p. 1

Dataset contribution

• HOT3D provides 1.5M multi-view frames and 3.7M+ images from 19 subjects interacting with 33 rigid objects across kitchen, office, living-room, and inspection scenarios.The dataset contains over 833 minutes of egocentric video from Project Aria and Quest 3.Source: §1 · p. 2, Figure 2 · p. 2

Annotations and assets

• Ground-truth annotations include headset, hand, and object poses, while object models provide high-resolution geometry and physically based rendering materials.Aria recordings additionally include scene point clouds and eye-gaze signals.Source: §1 · p. 2

Dataset capabilities

• Hardware-synchronized multi-view streams support temporal and multi-view methods for 3D hand-object tracking rather than only monocular or single-image analysis.The dataset also supports CAD-based and few-shot CAD-free object tracking, reconstruction, detection, segmentation, and gaze-based research.Source: §1 · p. 2

Empirical findings

• Multi-view methods significantly outperform single-view variants for 3D hand tracking, model-based 6DoF object pose estimation, and 3D lifting of unknown in-hand objects.These results motivate multi-view sensing for power-efficient egocentric vision systems.Source: §1 · p. 2

Dataset contribution

• HOT3D is presented as the first dataset combining real-headset, egocentric, hardware-time-synchronized multi-view videos with high-quality 3D hand and object annotations.Its total image count is reported as the largest among datasets with 3D hand and object annotations.Source: §1 · p. 2, Table 1 · p. 3

Hands and objects

• Earlier hand datasets progressed from monochrome and RGB-D imagery toward RGB or monochrome data, while object datasets similarly moved from monochrome and RGB-D benchmarks toward RGB-only pose estimation.Examples include NYU, ICVL, MSRA, UmeTrack, BOP datasets, OnePose, and HANDAL.Source: §2 · p. 3
• Many hand-object datasets provide only 2D boxes, masks, or action labels, and several 3D hand datasets omit 6DoF object poses.Earlier fully annotated examples used manual labeling, magnetic sensors, or synthetic images.Source: §2 · p. 3
• Existing joint hand-object datasets vary widely in scale and realism, including exocentric RGB-D, synthetic, static-grasp, single-view, and articulated-object settings.HO-3D, H2O, DexYCB, HOI4D, and ContactPose each address subsets of the broader problem with distinct constraints.Source: §2 · p. 3
• ARCTIC uses marker-based capture but shows articulated-object interactions from mostly exocentric views, with only one view recorded using a helmet-mounted mock egocentric device.It contains 2.1M RGB images from 10 subjects interacting with 11 articulated objects.Source: §2 · p. 4
• HO-Cap is another real-headset dataset with 3D annotations for both hands and multiple objects, but its RGB-D-camera annotations are less accurate than precise motion-capture annotations.This distinction separates headset-based capture from annotation accuracy.Source: §2 · p. 4

Data streams

• 833 minutes of synchronized Aria and Quest 3 recordings provide 1.5M+ multi-view frames and 3.7M+ images, with camera parameters and Aria scene point clouds and eye gaze.The streams are recorded at 30 fps; Aria contributes one RGB and two monochrome images per frame, while Quest 3 contributes two monochrome images.Source: §3 · p. 4

Dataset diversity

• The dataset covers 19 diverse subjects, 33 scanned household and office objects, and four scenarios spanning inspection, kitchen, office, and living-room actions.The object models include high-resolution geometry and PBR materials, while participants’ hands are provided in UmeTrack and MANO formats.Source: §3 · p. 4

Ground-truth annotations

• Per-frame hand and object poses are represented in UmeTrack, MANO, and rigid-transformation formats and obtained using marker-based motion capture with infrared OptiTrack cameras.The markers were registered to scanned hand and object models, supporting scene-space pose annotations.Source: §3 · p. 4, Figure 5 · p. 5
• 1.16M of 1.5M frames are fully annotated and visually validated, while all frames are released with a validity mask for filtering or unsupervised training.Some frames may have missing or lower-quality annotations, so the released mask identifies the inspected valid subset.Source: §3 · p. 4

Benchmark resources

• Training annotations cover 13 subjects, whereas the remaining 6-subject test split provides ground truth only through dedicated evaluation servers.This separation supports held-out-subject evaluation while restricting direct access to test annotations.Source: §3 · p. 4
• HOT3D releases 3832 five-second clips that passed annotation, visibility, exposure, and visual-quality checks for benchmarking tracking and pose-estimation methods.The clips are distributed across training and test splits and include recordings from both headsets.Source: §3 · pp. 4–5
• Object onboarding supplies static and dynamic reference sequences for model-free tracking and reconstruction, with full static poses but only the first dynamic pose defining canonical object space.This distinction reflects the easier pose recovery of static setups and the greater challenge of dynamic hand manipulation.Source: §3 · p. 5

4.1. 3D hand pose tracking

• 41% improvement in two-view hand tracking reaches 9.5 MKPE on UmeTrack and 10.9 on HOT3D when training combines both datasets.Single-view performance is 13.4 MKPE on UmeTrack and 15.4 on HOT3D; combining datasets closes the domain gap.Source: §4.1 · p. 5, Table 2 · p. 6

4.2. 6DoF object pose estimation

• 8–12% higher recall rates, or 13–34% relative improvement, result from the multi-view FoundPose extension on both headsets.The extension adds multi-view 2D-3D constraints and can exploit views that reveal heavily or fully occluded objects.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. 2D segmentation of in-hand objects

• MRCNN-DA outperforms EgoHOS by 30% on Aria and 65% on Quest 3 for in-hand object segmentation.Predicted depth improves foreground-object versus background disambiguation, while EgoHOS suffers a domain gap, especially on Quest 3 monochrome images.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. 3D lifting of in-hand objects

• StereoMatch triangulates DINOv2 feature correspondences across stereo crops, while MonoDepth aligns monocular predictions with SLAM point clouds.MonoDepth is evaluated only on Aria because Quest 3 lacks the required SLAM observations.Source: §4.4 · p. 8
• StereoMatch outperforms MonoDepth with either ground-truth or predicted masks and is strongest at stricter 3D localization thresholds.HandProxy suffices within a 30cm uncertainty radius, whereas dedicated multi-view lifting adds value for localization within 10cm.Source: §4.4 · p. 8, Table 5 · p. 8

5. Conclusion

• HOT3D enables benchmarking 2D and 3D egocentric hand-object tasks, including multi-view object tracking, 6DoF pose estimation, and 3D lifting.The dataset and associated public challenges are released to support research on egocentric vision and contextual AI.Source: §5 · p. 8

Appendix

• The appendix covers Aria and Quest 3 devices, ground-truth annotation, additional data statistics, and quantitative results.These materials are organized across Sections A–E.Source: Appendix · p. 11

A. Aria glasses

• Project Aria is a Meta research device in glasses form factor designed for egocentric machine perception and contextualized AI research.It is available to researchers through projectaria.com.Source: A. Aria glasses · p. 11

• Project Aria weighs 75g and is designed for prolonged wear without obstructing natural motion during dynamic activities.Source: §A.1 · p. 11
• Its HOT3D sensor profile combines one RGB camera, two monochrome cameras, two eye-tracking cameras, two IMUs, a barometer, and a magnetometer.The cameras operate at 30fps or 10fps, while the IMUs operate at 800Hz and 1000Hz.Source: §A.1 · p. 11
• The RGB and monochrome cameras provide wide-field egocentric visual coverage, while the eye cameras capture gaze-related imagery.Source: §A.1 · p. 11, Figure 11 · p. 12
• GNSS, WiFi scanning, and audio recording were disabled for HOT3D, with audio omitted for privacy reasons.Source: §A.1 · p. 12

• MPS provides calibrated software building blocks for using Project Aria’s multimodal recordings, with raw outputs released as part of HOT3D.Source: §A.2 · p. 12
• Aria localization uses VIO and SLAM to produce metric, gravity-aligned 6DoF camera poses for every frame and 1kHz motion between frames.Source: §A.2 · p. 12
• The gaze service represents viewing direction as two outward-facing rays and can personalize estimates through an optional wearer calibration procedure.Source: §A.2 · p. 12
• MPS outputs include RGB views with gaze, reconstructed point clouds, and estimated camera trajectories for recordings in multiple environments.Source: Figure 12 · p. 12
• The point-cloud service causally triangulates static scene elements from moving-device imagery and retains their 3D locations with source 2D observations.Source: §A.2 · p. 12

A.3. Processing summary

• 199 Aria recordings totaling 391 minutes were anonymized, processed through MPS, and 7DoF-aligned with the OptiTrack coordinate frame.Source: §A.3 · p. 12

A.4. Tools and ecosystem

• Open-source Python and C++ tools support loading, conversion, visualization, and common computer-vision workflows for Aria data.Source: §A.4 · p. 13

B. Quest 3 headset

• Quest 3 contributes synchronized images from two front global-shutter monochrome cameras, whose calibration and per-frame camera-to-world transforms are obtained through marker tracking.Source: B. Quest 3 headset · p. 13

C. Marker-based motion capture

• Optical markers on hands and objects are registered to scanned 3D models, enabling UmeTrack hand fitting and object-pose estimation in model coordinates.Source: C. Marker-based motion capture · p. 13
• SMPTE timecode synchronizes data frames from the different recording sources.Source: C. Marker-based motion capture · p. 13

• Natural interaction produces clear object-specific pose biases in the observed orientation distributions.Source: D. Object orientation statistics · p. 13
• Object orientation statistics report azimuth from 0° to 360° and elevation from −90° to 90° across observed dataset poses.Source: Figure 15 · p. 13
• The figure pairs canonical 3D object models with distributions of azimuth and elevation angles observed across the dataset.Source: Figure 15 · p. 13
• The bowl tends to appear upright, while the birdhouse tends to appear from the front and upright, providing potentially useful inference priors.Source: D. Object orientation statistics · p. 13

• Training-split evaluation is possible because both tasks are training-free and do not require a training split.Source: E. Additional quantitative results · p. 13
• Training+test, training, and test splits are reported for 2D segmentation of in-hand objects.The corresponding table reports mIoU scores for each split.Source: E. Additional quantitative results · p. 13, Table 6 · p. 14
• Training+test, training, and test splits are reported for 3D lifting of in-hand objects.The corresponding table reports recall rates for each split.Source: E. Additional quantitative results · p. 13, Table 7 · p. 14