EgoPhys: Estimating Peak Contact Force and Mechanical Work from Egocentric Manipulation Video

1. INTRODUCTION

• Prior methods recover forces from motion and dynamics, estimate pressure from RGB [7, 8], or use RGB-D instructions and tactile deformation [10], leaving egocentric RGB force-and-work estimation underexplored.Earlier articulation methods localize actionable regions, infer geometry and motion [3], or learn policies [4], but provide limited information about interaction quantities.Source: §1 · p. 1
• Hoi! recordings pair head-mounted egocentric video with synchronized force/torque and tool-motion measurements during articulated manipulation.Other egocentric datasets lack force sensing, while cited force-recording datasets primarily target robot learning.Source: §1 · p. 1
• Peak force depends on brief high-load events, whereas mechanical work depends on force–motion coupling throughout contact.Interaction cues are subtle, localized near gripper–object contact, and can be overwhelmed by background content and camera motion.Source: §1 · pp. 1–2
• EgoPhys combines CASA, which aggregates interaction-relevant appearance and geometry, with TMTR, which routes semantic, event, and motion features separately for force and work.The framework predicts peak contact force and contact-phase mechanical work from egocentric RGB video alone.Source: Figure 1 · p. 1, §1 · p. 2
• The authors investigate joint RGB-only estimation of peak contact force and contact-phase mechanical work, with experiments on a fixed Hoi! test split demonstrating effectiveness for both targets.The introduction presents this joint task and the framework as its contribution.Source: §1 · p. 2

2.1. Problem Formulation

• The task predicts peak resultant contact force and contact-phase work from RGB clips, with contact defined where force magnitude exceeds 2 N.Peak force is measured in newtons and work in joules.Source: §2.1 · p. 2

2.2. Framework Overview

• EgoPhys combines CASA’s interaction-relevant spatial aggregation with TMTR’s separate temporal modeling for force and work prediction.CASA fuses appearance and geometry features, while TMTR models semantic, event, and motion streams with separate temporal experts.Source: §2.2 · p. 2

2.3. Contact-Aware Spatial Aggregation

• CASA cross-attends between appearance and geometry features, then scores spatial locations to form a weighted frame-level representation.Dense features from DINOv3 and Depth Anything 3 are projected into a common space.Source: §2.3 · p. 2

2.4. Target-Specific Multi-Expert Temporal Routing

• Three experts model semantic context, brief force-indicative events, and sustained work-related motion; target-specific gates mix their outputs across multiple temporal scales.The motion expert uses differences at lags 1, 4, and 8; expert blocks use dilation rates 1, 2, 4, 8, 16, and 32.Source: §2.4 · pp. 2–3

2.5. Prediction and Learning Objectives

• Separate regression heads produce nonnegative force and work estimates, while training adds frame-level mean-force and loaded-interval supervision to the regression loss.The auxiliary losses are masked Smooth L1 for mean force and masked binary cross-entropy for the loaded interval.Source: §2.5 · p. 3

3.1. Experimental Setup

• The dataset comprises 336 clips from 22 gripper recordings across 18 scenes, split into 264 training, 36 validation, and 36 test clips.Evaluation uses MAE, MedianAE, RMSE, and MRE; results are averaged over six random seeds.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2. Experimental Results

• 5.205 N vs. 8.796 N force MAE and 0.894 J vs. 1.624 J work MAE; EgoPhys had the lowest errors across all four metrics for both targets.The comparison is with DINOv3+DA3.Source: §3.2 · p. 4, Table 1 · p. 4

3.3. Ablation Study

• CASA lowers force/work MAEs to 7.776 N/1.456 J; TMTR reaches 5.982 N/1.103 J, and auxiliary losses reach 5.205 N/0.894 J.The progression starts from the DINOv3+DA3 baseline at 8.796 N and 1.624 J.Source: §3.3 · p. 4, Table 2 · p. 4
• The semantic expert is strongest alone, but gated routing beats direct concatenation: force MAE 5.205 N vs. 6.303 N and work MAE 0.894 J vs. 1.189 J.The authors conclude that selectively combining temporal cues is more effective than simple concatenation.Source: §3.3 · p. 4, Table 3 · p. 4

• 5.205 ± 0.584 N peak-force MAE and 0.894 ± 0.081 J work MAE on the fixed Hoi! test split, improving over DINOv3+DA3.The conclusion also reports that ablations support the effectiveness of CASA and TMTR.Source: §4 · p. 4