Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation

2. Related Work

• Existing resources often separate long-horizon human activity from short-horizon robotic primitives and lack paired motion, force, tactile, viewpoint, or embodiment data.This limits systematic study of how visual appearance relates to physical interaction and how human skills transfer to robots.Source: §1 · pp. 1–2, §2 · pp. 2–3

1. Introduction

• Hoi! combines human and robotic interactions with the same articulated objects across vision, force, tactile sensing, and multiple viewpoints.The four embodiments include a human hand, wrist-camera hand, UMI gripper, and sensorized Hoi! gripper.Source: §1 · p. 2, §2 · p. 4
• The dataset supports benchmarks in force-from-vision, articulation estimation, cross-view transfer, and state-change prediction.Dense scene scans, object metadata, joint parameters, and sequence-level state-change labels anchor interaction understanding to geometry.Source: §1 · p. 2
• 3048 sequences cover 381 articulated objects in 38 environments and provide time-aligned vision, pose, force, haptic streams, and scene-level ground truth.The dataset spans four main embodiments and supports interaction-centric annotations.Source: §1 · p. 2

3. Hoi! Dataset

• Hoi! records the same articulated objects across human-hand, wrist-camera, UMI-gripper, and Hoi!-gripper embodiments with synchronized egocentric and third-person views.A subset also uses a teleoperated Spot robot, while all modalities are temporally and spatially aligned.Source: §3 · p. 4, Figure 7 · p. 6
• The custom Hoi! gripper measures gripping force, wrist interaction forces, and tactile contact while providing pose and wrist-view observations.It combines Digit tactile sensors, a 6-DoF force–torque sensor, a calibrated load cell, stereo vision, and Aria tracking.Source: §3 · p. 4, Figure 3 · p. 4
• The collection protocol spans furniture-store, laboratory, and apartment environments, with seven demonstrators recording 3–11 articulated parts per session.Dynamic and static QR codes synchronize streams and mark individual interactions.Source: §3 · p. 4
• The dataset includes diverse environments, articulated categories, viewpoints, trajectories, contacts, haptic feedback, force measurements, and 3D scene data.These streams support direct comparison across human and robot executions.Source: Figure 5 · p. 5, Figure 6 · p. 6, Figure 7 · p. 6
• High-resolution before/after laser scans provide ground-truth geometry and a shared reference frame for spatial calibration.Each location receives 2–5 point clouds capturing unarticulated and opened configurations.Source: §3 · p. 5, Figure 6 · p. 6
• QR-code timestamping aligns independent recording clocks to about 10–25 ms, while visual localization aligns sensor trajectories spatially.Trajectory accuracy is evaluated against Qualisys motion capture.Source: §3 · p. 5, Table 3 · p. 6

4. Evaluations

• The evaluation covers articulation estimation, tactile force estimation from GelSight images, and visual prediction of interaction forces and affordances from RGB-D observations.These tasks use aligned poses, forces, tactile signals, language goals, and scene geometry.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 7

4.1. In-the-Wild Articulated Object Estimation

• Current articulation estimators underperform in-the-wild, because clutter, unreliable segmentation, and noisy monocular depth limit 3D lifting and trajectory filtering.GPT-5 remains surprisingly robust for articulation-type prediction from single RGB views.Source: §4.1 · p. 7, Table 4 · p. 7

4.2. Tactile Force Estimation

• Sparsh tactile-force prediction produces errors of several Newtons on Hoi!, versus millinewton-level accuracy on its original benchmark.The authors attribute degradation to out-of-distribution contact geometries and load regimes in real furniture interactions.Source: §4.2 · p. 7, Table 5 · p. 8

4.3. Visual Force Estimation

• ForceSight reaches 0.404 N RMSE on its original dataset but degrades on Hoi!, reaching 3.531 N in kitchen_7 and 2.325 N in office_1.Projected forces and torques improve performance relative to raw measurements, indicating sensitivity to operator-induced disturbances.Source: §4.3 · p. 8

• Human-operated Hoi! gripper demonstrations do not fully capture real manipulators’ kinematic and dynamic constraints, leaving full-body morphology generalization open.Source: §5 · p. 8
• The dataset spans many articulated household objects but omits some mechanical complexities and rare edge-case mechanisms.Source: §5 · p. 8
• Current benchmarks focus on object-centric perception tasks, while end-to-end policy learning remains an important future direction.Source: §5 · p. 8

6. Conclusions

• Hoi! bridges human-centric and robot-centric interaction data with a gripper that captures force, torque, tactile, and visual signals during real-world manipulation.Source: §6 · p. 8
• The dataset supports object-centric inference of affordances, articulated motion, and physical effort across tactile force estimation, motion prediction, and visual force estimation.Source: §6 · p. 8
• The work contributes a richly annotated, cross-embodiment and cross-view dataset for agents that observe, understand, and act upon the physical world.Source: §6 · p. 8

7. Funding & Acknowledgments

• The research was partially funded by the ETH AI Center, ETH Foundation Project 2025-FS-352, SNSF Advanced Grant 216260, and the Lamarr Institute.Source: §7 · p. 9

4. Evaluations

• The listed evaluation sections cover in-the-wild articulation estimation, tactile force estimation, visual force estimation, multimodal learning, and hand pose estimation.Source: §4 · p. 12

7. Funding & Acknowledgments

• The supplementary listings include dataset details, gripper calibration, sensor alignment, and time and spatial alignment sections.Source: §7 · p. 12

D. Evaluations

• The supplementary material lists evaluation subsections and dataset-detail sections, but the supplied passages do not describe Spot recordings.Source: D. Evaluations · p. 12, A. Hoi! Dataset Details & Gripper… · p. 12, §A.1 · p. 12

A.1. Recording Device Specs

• Force-torque measurements have below 2% accuracy across axes and 70–100 mN / 0.6–2.1 mNm noise-free resolution at 100 Hz.Source: §A.1 · p. 12
• The Hoi! gripper measures interaction force and tactile contact pressure through force-torque and Digit sensors, while cameras provide pose and wrist-view observations.Source: Figure 8 · p. 12

A.2. Motor Calibration

• The gripping-force calibration maps motor current through the gripper Jacobian and a load-dependent efficiency factor.Source: §A.2 · pp. 12–13
• The efficiency factor is estimated by least-squares fitting measured gripping forces and motor currents across multiple load regimes.Source: §A.2 · p. 13

A.3. Inter-Sensor Calibration

• Camera intrinsics, visual-inertial extrinsics, and stereo-camera-to-force–torque calibration establish the gripper’s inter-sensor geometry.Calibration proceeds from ZED intrinsics through IMU alignment and finally stereo calibration with the force–torque sensor.Source: §A.3 · p. 13

A.4. Gripper Gravity Compensation

• Gravity compensation separates external interaction forces from gravitational loads and internal force–torque sensor biases.The method uses measured mass, CAD-derived center of mass, world-frame gravity, and no-load bias estimates.Source: §A.4 · p. 13
• No-load windows are detected from filtered force magnitudes, and least squares estimates the sensor biases before external forces and torques are computed.A fourth-degree Butterworth filter is subsequently applied to reduce noise.Source: §A.4 · p. 13

B. Alignment of Sensors in the Hoi! Dataset Recordings

• Temporal and spatial alignment place the multimodal recordings into shared coordinates so interactions can be captured across time and viewpoints.The supplied passages describe this alignment as part of the dataset recording pipeline.Source: B. Alignment of Sensors in the Hoi!… · p. 13

B.1. Time Alignment

• QR codes encoding Unix timestamps provide post-processing offsets for independently clocked video streams relative to the Aria reference.The QR code is displayed at 25 Hz in each camera stream.Source: §B.1 · pp. 13–14
• 10–25 ms temporal alignment accuracy is achieved for typical 30–60 Hz frame rates, with approximately 10.8 ms uncertainty for 60 Hz iPhone and 30 Hz Aria recordings.The uncertainty is attributed to frame-timing quantization in the first detected QR-coded frame.Source: §B.1 · p. 15

B.2. Spatial Alignment

• Spatial alignment localizes device keyframes against dense 2D–3D correspondences from high-resolution 3D scans and estimates rigid transformations into a common world frame.Keyframes are filtered for feature density, sharpness, and scene depth before DINOv2-based diversity sampling and hierarchical localization.Source: §B.2 · p. 15
• Outlier-rejected transformation averaging produces globally aligned 6-DoF trajectories for Aria, hands, and the Hoi! gripper, while statically mounted iPhone and transformed UMI poses are also registered.The alignment uses a single robust device-to-world transformation after pairwise consistency checks.Source: §B.2 · p. 16

C. Spot Recordings

• Spot is teleoperated with a Meta Quest 3, retargeting the remote’s 6-DoF pose to the gripper and using its joystick for base control.The Spot recordings include robot joint states, RGB-D cameras, gripper imagery, and wrist-mounted Aria data.Source: C. Spot Recordings · p. 16, Figure 12 · p. 16

D.1. In-The-Wild Articulation Estimation

• Articulation estimation uses posed egocentric and exocentric RGB frames, annotations, and 3D ground truth, with MapAnything depth rescaled by the mode of predicted-to-rendered depth ratios.The resulting depth is metrically aligned for evaluating articulation parameters and types.Source: §D.1 · p. 16

D.2. Tactile Force Estimation

• Tactile force estimation combines GelSight images with force–torque and gripping-force labels, evaluating Sparsh decoders for normal, tangential, and combined forces.Ground-truth forces are expressed in an interaction frame aligned with the Digit sensors, while internal gripping force is estimated from motor behavior.Source: §D.2 · p. 17
• The tactile models generally under-predict forces even when ground-truth magnitudes are clipped to the training range.The evaluation uses clipping to avoid extrapolation to unseen force magnitudes.Source: §D.2 · p. 17

D.3. Visual Force Estimation

• Visual force estimation combines an RGB-D observation and an open-language manipulation goal to predict where to interact and what force to apply.The evaluation uses ForceSight across a diverse subset of six environments.Source: §D.3 · p. 17
• The evaluation reports both raw and motion-aligned force–torque signals, projecting forces and torques onto linear and rotational gripper velocities.Motion alignment removes components unrelated to the articulated interaction for a fairer evaluation signal.Source: §D.3 · p. 17

D.4. Multimodal Learning for Real-World Robotics

• The force predictor combines frozen DINOv2 features, RGB-D observations, and an articulated-part class label to regress log-transformed peak interaction force.A learned residual is estimated over a class-specific force prior, with the visual residual scaled by α.Source: §D.4 · p. 18
• The predicted force prior is mapped back to force space and added as a feed-forward term to standard impedance control using pose and velocity errors.The position-force variant augments the baseline controller rather than replacing its impedance terms.Source: §D.4 · p. 18
• 100%, 80%, and 60% position-force success rates are reported for the drawer, oven, and dishwasher, compared with 100%, 0%, and 0% for position control.Average predicted forces are 16,80 N, 46,38 N, and 53,29 N for the drawer, oven, and dishwasher, respectively.Source: §D.4 · pp. 18–19

D.5. Hand Pose Estimation

• Hand-pose evaluation is exploratory because Aria MPS keypoints are automatically generated, while egocentric performance trails controlled benchmarks under fast motions, natural behaviors, and lower-resolution imagery.The authors frame this gap as evidence of challenging real-world egocentric scenes and an opportunity to exploit multimodal signals.Source: §D.5 · p. 19