PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes

Hongming Fu, Jingcheng Shi, Wenjia Wang, Binhua Zuo, Bo Zhao

arXiv:2610.01210v1cs.CV

TL;DR

Egocentric video is valuable for embodied-model supervision, but recovering world-space hand motion is difficult and existing pipelines separate reconstruction from interaction prediction. EgoFound3R combines structured prompts, an explicit hand representation, and multi-rate processing to predict geometry and point-wise attributes in one pass. It lowers MPJPE across three datasets and runs approximately 6× faster than WiLoR.

  • Problem

    World-space hand recovery from egocentric video is difficult because camera motion and hand–object occlusion corrupt monocular cues, while existing pipelines separate reconstruction and interaction prediction.

    Source: §1 · p. 2

  • Method

    EgoFound3R uses structured prompts, an explicit hand representation, and shared-parameter multi-rate processing to predict world-space hand geometry and point-wise attributes in one pass.

    Source: §1 · p. 2

  • Results

    43.2%, 22.4%, and 11.6% lower MPJPE on OakInk-v2, TACO, and HOI4D, respectively, with approximately 6× higher throughput than WiLoR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model produces world-space geometry and point-wise interaction attributes together at annotation-scale throughput.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

Egocentric video has become a primary source of supervision for embodied models, and its value rests on recovering hand motion in world coordinates, which camera motion and hand occlusion make difficult. Existing reconstruction pipelines typically separate hand and scene estimation, leave interaction attributes to separate task-specific models, and invoke several models per video, so no prior reconstruction model estimates these attributes and throughput becomes a practical constraint on large-scale annotation. We therefore introduce EgoFound3R, a unified end-to-end model that estimates world-space hand geometry in a metric scale shared with the scene, and predicts point-wise interaction attributes, including visibility, contact, and distance. The model integrates three designs: (i) structured hand prompts that transfer pretrained geometric priors to world-space hand reconstruction; (ii) an explicit hand representation that decodes hand geometry and interaction attributes; and (iii) a shared-parameter multi-rate design that lowers inference cost. Together, these designs predict hand geometry and point-wise attributes in one pass. On OakInk-v2, TACO, and HOI4D, EgoFound3R reduces the mean per-joint position error (MPJPE) by 43.2%, 22.4%, and 11.6% over previous methods and predicts point-wise contact and distance alongside the geometry in the same pass, while attaining approximately 6x higher throughput.

1 INTRODUCTION

EgoFound3R unifies world-space hand reconstruction and point-wise interaction-attribute prediction in one model, addressing the accuracy and throughput constraints of egocentric-video supervision. It lowers MPJPE on three datasets and achieves approximately 6× higher throughput than WiLoR.

1 INTRODUCTION

  • Egocentric recordings capture manipulation as performed and are increasingly used to train embodied models.Source: §1 · p. 1
  • Camera egomotion, hand–object occlusion, and absent absolute scale make world-space hand recovery difficult; staged pipelines also accumulate errors and reduce throughput.Existing reconstruction models leave visibility, contact, and distance to separate predictors.Source: §1 · p. 2
  • Structured prompts condition an explicit hand representation on a frozen geometry backbone, while shared-parameter multi-rate processing reduces redundant scene aggregation.The model decodes geometry and attributes from shared point features and uses an observation-constrained root estimate to place the hand in the scene.Source: §1 · p. 2
  • 43.2%, 22.4%, and 11.6% lower MPJPE on OakInk-v2, TACO, and HOI4D, respectively, with approximately 6× higher throughput than WiLoR.Source: §1 · p. 2
  • EgoFound3R reconstructs world-space hands and predicts point-wise visibility, contact, and distance in one pass, replacing a chain of separate estimation stages.Source: §1 · p. 2

2 RELATED WORK

Related work develops geometric reconstruction priors, egocentric-data annotation pipelines, hand reconstruction methods, and interaction predictors. EgoFound3R combines these directions by decoding world-space hand geometry and point-wise interaction attributes from shared features.

Geometric priors and unified reconstruction

  • EgoFound3R extends joint hand-and-scene reconstruction by adding interaction attributes to shared full-image features without a separate detection or crop stage.Related systems include Human3R (Chen et al., 2026b) and ReViV (Lyu et al., 2026).Source: §2 · p. 3

Egocentric data for embodied models

  • Egocentric-data systems increasingly support embodied-model training, making annotation throughput a practical constraint addressed by EgoFound3R’s single-model outputs.Source: §2 · p. 3

Hand geometry and motion

  • Prior hand methods estimate camera-space or world-space geometry, refine motion, or regress mesh coordinates; EgoFound3R attaches point-wise interaction attributes to an explicit representation.Source: §2 · p. 3

Contact and interaction data

  • Interaction methods use geometric or visual cues for contact and hand–object interaction, while annotation systems may chain several learned models per video.EgoFound3R replaces that chain with one model and requires no object meshes at inference.Source: §2 · p. 3

3 METHOD

EgoFound3R combines dense hand observations with sparser scene anchors to reconstruct world-space hands and decode point-wise interaction attributes. It uses pretrained geometry features, shared point decoding, metric scale alignment, and post-network output processing.

3.1 PROBLEM FORMULATION

  • Hand geometry and point-wise attributes are predicted at every hand time H, while sparse scene anchors G provide depth and camera motion; outputs are returned at requested times R.Source: §3.1 · pp. 3–4

3.2 HAND PROMPTS THAT REUSE PRETRAINED GEOMETRY

  • Root, joint, and surface prompts query frozen VGGT-Ω features with scene context; a bidirectional temporal Transformer links hands, while the backbone remains frozen.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 CROSS-RATE INTERACTION BETWEEN HANDS AND SCENE

  • Stride s pools intervening hand prompts into lower-rate scene anchors; zero-initialized residual updates preserve dense behavior initially, and shared adapters let the ratio vary without a new model.Source: §3.3 · pp. 4–5

3.4 EXPLICIT HAND GEOMETRY WITH INTERACTION ATTRIBUTES

  • Hand points are decoded directly from shared features, with each point receiving visibility, contact, and distance attributes rather than using pose or shape coefficients.Source: §3.4 · p. 5, Figure 4 · p. 5

3.5 METRIC HAND PLACEMENT

  • A wrist-constrained solve combines 2D rays, relative joints, and a learned depth prior for root placement; calibration rescales scene depth and camera translation, not rotations or camera-space hands.Source: Figure 5 · p. 5, §3.5 · pp. 5–6

3.6 TRAINING AND OUTPUTS AT ARBITRARY TIMES

  • Post-network processing interpolates requested times, densifies 195-vertex geometry to 778-vertex MANO, and scale-aligns overlapping clips; future-frame interpolation makes the method offline.Source: Figure 5 · p. 5, §3.6 · p. 6

4 EXPERIMENTS

EgoFound3R is evaluated on three benchmarks for world-space hand geometry, interaction attributes, qualitative reconstruction, design choices, and inference efficiency. It improves hand-geometry accuracy, predicts contact and visibility alongside geometry, and processes hand observations faster than the compared pipelines.

4.1 EVALUATION SETTING

  • Evaluation uses 2,378 held-out two-second windows sampled at a fixed ratio per dataset; the main tables report OakInk-v2, TACO, and HOI4D.Training uses seven egocentric datasets; Dyn-HaMR and InteractVLM are evaluated on matched 100-window subsets.Source: §4.1 · p. 6

4.2 HAND GEOMETRY AND INTERACTION

  • 43.2%, 22.4%, and 11.6% MPJPE reductions over the strongest baseline on OakInk-v2, TACO, and HOI4D, respectively.The reductions are reported for world-space hand reconstruction; the ordering also survives removal of root translation.Source: §4.2 · p. 6, Table 1 · p. 7
  • EgoFound3R attains the highest contact precision, recall, and F1 on all three subsets, while also predicting per-point distance to the contact surface.The compared systems use different contact conventions, and the two geometric baselines receive ground-truth geometry.Source: Table 2 · p. 7, §4.2 · p. 7
  • EgoFound3R surpasses HVD in joint-visibility F1 on all three datasets and also predicts vertex-level visibility.It has higher precision on two subsets and higher recall on two others.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 WORLD-SPACE QUALITATIVE COMPARISON

  • In the world-frame comparisons, EgoFound3R reproduces the ground-truth hand layout with its predicted cameras, while camera-space estimators offset hands from objects and world-space pipelines drift.Figure 6 shows two clips from two fixed viewpoints, including hand placement relative to objects.Source: §4.3 · pp. 7–8, Figure 6 · p. 8

4.4 CAMERA-SPACE QUALITATIVE COMPARISON

  • Across four held-out windows, EgoFound3R tracks the annotated hand and its visibility, contact, and distance maps follow the ground truth.The comparisons are 2D overlays on observed pixels, with hand geometry above the attribute maps.Source: Figure 7 · p. 8, §4.4 · pp. 8–9

4.5 CORE DESIGN ABLATION

  • On TACO, reducing hand-module capacity, removing the frozen VGGT-Ω backbone, restricting hand–scene exchange to one direction, or replacing point decoding with MANO regression degrades reported outcomes.The stride ablation for other rates is reported in Appendix I; S5 is the default configuration.Source: §4.5 · p. 9, Table 4 · p. 9

4.6 INFERENCE EFFICIENCY

  • 28.69 H FPS at 512 × 512 is 5.97× the 4.81 FPS of WiLoR, with one forward pass determining throughput.H FPS measures hand-observation frames processed per second; Fig. 8 and Table 6 also report dense output rate.Source: Figure 8 · p. 9, §4.6 · p. 9

5 CONCLUSION

EgoFound3R combines world-space hand reconstruction with point-wise visibility, contact, and distance prediction in one pass. The appendix details how its prompts, multi-rate information exchange, decoding heads, and scene-scale estimation implement that design.

5 CONCLUSION

  • EgoFound3R reconstructs hands in world space and predicts point-wise visibility, contact, and distance in one pass, reducing MPJPE by 43.2%, 22.4%, and 11.6% on OakInk-v2, TACO, and HOI4D while running 6× faster than the previous state of the art.Its design combines structured prompts, an explicit hand representation, and shared-parameter multi-rate processing.Source: §5 · p. 9

APPENDIX CONTENTS

  • Appendix sections cover training objectives, inference efficiency and temporal rates, evaluation metrics, contact and visibility evaluation, and matched-subset controls.The contents also list additional geometry evaluation, architecture and multi-rate ablations, and camera- and world-space qualitative results.Source: APPENDIX CONTENTS · p. 13

A DETAILED ARCHITECTURE AND INFORMATION FLOW

  • The appendix details hand prompts and localization, cross-rate information exchange, explicit geometry and attribute decoding, scene and metric heads, and requested-time output processing.Requested times are produced by a parameter-free stage that interpolates cameras and hands separately; these operations use future frames and remain outside the feed-forward network.Source: A DETAILED ARCHITECTURE AND… · pp. 13–14

B TRAINING OBJECTIVES AND SUPERVISION

Training combines masked geometry, interaction-attribute, 2D, and scene/camera objectives. Inference comparisons vary temporal stride while holding model, resolution, and window settings fixed.

B TRAINING OBJECTIVES AND SUPERVISION

  • Valid masked targets across seven egocentric datasets contribute to a multi-task objective spanning geometry, visibility, contact, distance, 2D cues, and scene/camera supervision.Contact and distance targets use point-to-triangle distances to annotated object or opposite-hand surfaces; only finite targets with valid masks contribute.Source: B TRAINING OBJECTIVES AND SUPERVISION · pp. 14–15
  • Hand geometry is trained with confidence-weighted robust regression on root-relative offsets normalized by hand-point scale, while the root is recovered from metric scene and image observations.The same geometry-loss form applies to joints and native surface points.Source: B TRAINING OBJECTIVES AND SUPERVISION · p. 15
  • Visibility uses masked binary cross-entropy, contact classification couples its error to learned point confidence, and distance regression targets log-distance to retain near-contact resolution.The 2D localization branch combines heatmap BCE, Smooth-L1 offsets, and projection consistency; scene/camera branches supervise depth, intrinsics, and camera motion.Source: B TRAINING OBJECTIVES AND SUPERVISION · p. 15

C INFERENCE EFFICIENCY AND TEMPORAL RATES

  • Inference efficiency is compared on one NVIDIA H20; the stride ablation changes global stride alone, with parameters, 512×512 input, and 2 s windows fixed.The efficiency table reports anchor, hand, and output rates, throughput, VRAM, and parameter counts; ContactOpt and S2Contact are reported separately because they consume ground-truth geometry and object poses.Source: Figure 10 · p. 16, Table 6 · p. 16, C INFERENCE EFFICIENCY AND TEMPORAL… · pp. 16–17

D EVALUATION METRICS AND RANKING RULES

The evaluation defines alignment-specific geometry errors and per-point interaction metrics under a common window protocol. Additional sections compare hand geometry and temporal consistency across datasets and point resolutions.

D EVALUATION METRICS AND RANKING RULES

  • The common evaluation samples 2,378 disjoint 2 s windows from six datasets; Dyn-HaMR and InteractVLM are evaluated on matched 100-window subsets and ranked only within their pairs.Table 7 distinguishes training sequences from evaluation windows, and STERA-10M contributes training data only.Source: D EVALUATION METRICS AND RANKING RULES · p. 17, Table 7 · p. 17
  • Raw error uses window coordinates, RR subtracts wrist translation, PA aligns each frame, and Sim(3) applies one similarity transform per window.World-coordinate W and WA fit a Sim(3) on the first two valid frames or all valid hand points, respectively; camera-source conditions affect ranking groups.Source: D EVALUATION METRICS AND RANKING RULES · p. 17
  • Contact and visibility use per-point precision, recall, and F1 averaged across windows; distance MAE is reported over all points, predicted-contact points, and ground-truth-contact points.Threshold-sweep AP and best-F1 pool points by dataset, unlike the window-averaged F1@0.5.Source: D EVALUATION METRICS AND RANKING RULES · p. 18

E ADDITIONAL HAND GEOMETRY EVALUATION

  • Across OakInk-v2, TACO, and HOI4D, EgoFound3R leads unaligned, root-relative, and world geometry metrics at 21 joints and 195- and 778-vertex resolutions.The native 195 points use the fixed level-1 MANO sampling; 778-vertex results are recovered with the matching upsampling matrix.Source: E ADDITIONAL HAND GEOMETRY EVALUATION · p. 18, Table 8 · p. 18

F TEMPORAL EVALUATION

G CONTACT AND VISIBILITY EVALUATION

Contact and visibility are evaluated at point level, with contact labels tied to specified surface-distance tolerances. The section also examines threshold sensitivity, predicted-positive distances, and matched-subset comparisons.

G CONTACT AND VISIBILITY EVALUATION

  • Contact labels use distance thresholds of 18 mm at joints and 14 mm at 195 hand vertices, while ContactOpt and S2Contact use a narrower 2 mm coverage tolerance.The geometric baselines also receive ground-truth hand and object geometry.Source: G CONTACT AND VISIBILITY EVALUATION · pp. 19–20, Table 11 · p. 20
  • Vertex-level visibility is reported at native 195-vertex and interpolated 778-vertex resolutions, which the compared pipelines do not provide.HVD predicts joint visibility only, so it is not ranked for vertex-level visibility.Source: G CONTACT AND VISIBILITY EVALUATION · p. 20, Table 12 · p. 20
  • Across three datasets, EgoFound3R attains the highest AP and best-F1 in the threshold sweep, with best-F1 thresholds of 0.39–0.51 versus geometric-baseline peaks as low as 0.09.The sweep post-processes cached scores and distances without modifying models or predictions.Source: G CONTACT AND VISIBILITY EVALUATION · p. 20, Table 13 · p. 21
  • 57–65% of EgoFound3R’s predicted positives fall within 14 mm, with median distances of 10–12 mm; geometric-baseline medians span 10–21 mm.Figure 11 plots cumulative positive-distance fractions and ground-truth-contact recall as the labeling threshold varies.Source: G CONTACT AND VISIBILITY EVALUATION · p. 20, Figure 11 · p. 22

H MATCHED-SUBSET CONTROLS FOR 100-WINDOW BASELINES

I ARCHITECTURE AND MULTI-RATE ABLATIONS

The ablations vary hand-decoder capacity, representation, pretrained-backbone use, cross-rate exchange, and temporal stride to examine distinct design choices.

I ARCHITECTURE AND MULTI-RATE ABLATIONS

J SCENE AND CAMERA EVALUATION

On TACO, EgoFound3R has the lowest camera error among four hand-centric world-space methods and leads the depth metrics among methods that return depth. Its scene ablation finds dense aggregation best for camera error, with the default configuration second.

Metrics and scale protocol

  • Camera error uses window-level Sim(3) alignment for ATE and rotation error relative to the first valid frame; depth metrics use a per-window scale fit only for scale-ambiguous predictions.Metric-scale models keep scale one; the table’s metric gauge reports the scale ratio aggregated in log space, where 1 means metric scale.Source: J SCENE AND CAMERA EVALUATION · p. 27

Comparison

  • EgoFound3R attains the lowest camera error among four hand-centric world-space methods and, among the two that return depth, the lowest AbsRel and RMSE and highest δ1.Its depth is evaluated at its own scale, whereas ReViV requires a per-window scale fit; HaWoR and Dyn-HaMR return no depth.Source: J SCENE AND CAMERA EVALUATION · p. 27, Table 20 · p. 27

Ablation

  • Dense aggregation attains the lowest camera error in the TACO scene ablation, while the default configuration ranks second.The w/o Ω and 1-way rows share an uncoupled scene export, so their identical scene and camera columns are not independent scene predictions.Source: J SCENE AND CAMERA EVALUATION · p. 27, Table 21 · p. 27

K CAMERA-SPACE QUALITATIVE RESULTS

The qualitative comparisons pair hand geometry with point-wise visibility, contact, and distance predictions against ground truth. They span ARCTIC, OakInk-v2, and TACO, with the TACO geometry and attribute panels shown separately.

Qualitative comparisons

Reading the panels

  • Predictions are overlaid on input-frame pixels without additional 2D refinement or fitting, and attribute panels compare EgoFound3R with ground truth on the same points as the quantitative tables.The overlay comparison does not depend on camera extrinsics.Source: K CAMERA-SPACE QUALITATIVE RESULTS · p. 29

TACO

  • Figure 14 presents TACO hand geometry, while Figure 15 shows the matching per-point attributes for the same windows and order.The attribute figure columns include RGB input and EgoFound3R predictions and ground truth for visibility, contact, and distance.Source: Figure 14 · p. 30, Figure 15 · p. 31

L WORLD-SPACE QUALITATIVE RESULTS

Figures 16 and 17 add world-space qualitative comparisons to Figure 6, preserving its layout and extrinsics convention. Each figure shows two fixed viewpoints per clip and sampled RGB frames alongside one column per method.

Additional world-space results

Loading 2610.01210v1…