PaperlayerView exact revision on arXiv

Source-linked AI summary

ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild

Hongyu Ma, Hairong Qu, Shiqi Zhao, Yongsong Yang, Peng Yin

arXiv:2609.34817v2cs.CVcs.AI

TL;DR

Metric 3D hand reconstruction from egocentric stereo lacks an end-to-end model and an in-the-wild benchmark. ESTHER uses calibrated stereo, temporal reasoning, and a calibrated pseudo-labeling pipeline to build ESTHER3D. It achieves strong benchmark and external results, with reported robustness to missing views and preserved metric scale in monocular fallback.

  • Problem

    Egocentric stereo hand reconstruction lacks an end-to-end model and an in-the-wild benchmark, despite its metric 3D constraints for hand perception.

    Source: §1 · p. 1

  • Method

    ESTHER combines calibrated stereo and temporal encoders with query-gated decoding, trains on calibrated teacher pseudo-labels, and supplies labels for ESTHER3D.

    Source: §1 · pp. 1–2

  • Results

    ESTHER leads on ESTHER3D pose accuracy at 17.7 mm MPJPE, generalizes to HOT3D with 30.5 mm zero-shot MPJPE, and retains near-unchanged relative pose with one view removed.

    Source: §5.2 · pp. 6–7, §5.3 · p. 7

  • Takeaways & Limitations

    Stereo guidance binds apparent hand scale to metric depth, and the model preserves metric scale in monocular fallback while adapting to different rigs and modalities with minimal fine-tuning.

    Source: §1 · p. 2, §5.3 · p. 7, §6 · p. 8

    Streaming without detection and joint modeling of hands, objects, and bodies remain future work.

    Source: §6 · p. 8

Abstract

from arXiv · show

Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.

1. Introduction

Egocentric stereo provides metric 3D constraints suited to hand perception, but metric hand reconstruction from wearable stereo lacks both an end-to-end model and an in-the-wild benchmark. ESTHER and ESTHER3D address the model and data gaps with a calibrated labeling pipeline and a stereo-temporal reconstruction model.

Motivation

  • Egocentric stereo hand reconstruction lacks an end-to-end model and an in-the-wild benchmark, despite its relevance to robotics, AR, and VR.The setting seeks MANO pose and shape plus global wrist translation in metric 3D from short-baseline head-mounted stereo video.Source: §1 · p. 1

Contributions

  • ESTHER3D pairs in-the-wild stereo training sequences labeled by a calibrated pipeline with a motion-capture ground-truth test set.The pipeline combines 2D evidence, stereo geometry, and hand priors, and is validated against motion-capture ground truth.Source: §1 · pp. 1–2

Motivation

  • Wearable stereo requires handling missing triangulation anchors, frame-level detection dropouts, and image motion that mixes hand articulation with head motion.Blur and occlusion can erase a hand from one view, while interacting hands can be missed for entire frames.Source: §1 · p. 2

Contributions

  • ESTHER combines calibrated stereo geometry and temporal context to predict metric MANO pose, shape, and absolute wrist translation.Its stereo image encoder, masked temporal motion encoder, and query-gated fusion decoder are designed for wearable egocentric stereo.Source: §1 · p. 2

2. Related Works

Prior egocentric and hand-interaction datasets offer varied supervision, but wearable RGB stereo with metric MANO labels remains distinct. Existing hand and multiview methods leave gaps in metric scale, temporal consistency, and robustness to missing observations.

2.1. Egocentric and Embodied Datasets

  • ESTHER3D targets in-the-wild calibrated short-baseline RGB stereo with metric MANO labels and train/test protocols, unlike HOT3D’s monochrome wearable stereo and motion-capture labels.Other hand-object datasets include FPHA [7], H2O [17], HOI4D [21], OakInk [43], DexYCB [2], ARCTIC [6], AssemblyHands [27], and HOT3D [1].Source: §2.1 · p. 2

2.2. 3D Hand Pose and Mesh Estimation

  • Monocular mesh methods recover articulation but infer scale and metric depth from learned priors, while monocular-video world-space methods retain ambiguous scale.Temporal hand trackers include HTT [41], HandFormer [35], and UmeTrack [11]; HaWoR [48] and Dyn-HaMR [47] use monocular SLAM and offline optimization.Source: §2.2 · p. 2

2.3. Multiview, Stereo, and Depth-Based Reconstruction

  • Prior stereo and multiview approaches estimate sparse keypoints, use non-RGB sensing, or operate single-frame, whereas ESTHER targets temporally consistent metric MANO reconstruction from wearable RGB stereo.The setting does not assume visibility in both views or detection in every frame.Source: §2.3 · p. 2

3. Method

ESTHER is trained through a calibrated teacher-labeling pipeline that bootstraps additional in-the-wild labels, and its benchmark test labels are checked against motion capture. Its model combines calibrated stereo features, masked temporal reasoning, and query-gated decoding to produce metric hand estimates.

3.1. Calibrated Teacher Labeling Pipeline

  • The calibrated teacher converts stereo 2D evidence into metric MANO pseudo-labels using stereo calibration, hand priors, and two-stage optimization.It begins from triangulated keypoints and per-view WiLoR estimates; shape is optimized and fixed before per-frame pose and wrist refinement.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Ground Truth Validation

  • The teacher’s zero-shot estimates on the motion-capture test set reach 15.9 mm MPJPE, 9.3 mm PA-MPJPE, and 19.1 mm wrist error.The reported accuracy approaches the roughly 10 mm uncertainty in solving the glove measurements; heavy-occlusion 2D misdetections are handled with confidence thresholds and manual quality control.Source: §3.2 · p. 3

3.3. In-the-Wild Data and Filtering

  • ESTHER3D labels are filtered by geometric checks, stereo reprojection against detections, and human review before training and iterative teacher refinement.The first teacher pass labels 10 hours sampled across six categories, after which ESTHER predictions initialize teacher optimization on additional data.Source: §3 · p. 2, §3.3 · p. 3, §3.8 · p. 5

3.4. Model Overview

  • ESTHER’s stereo image, masked temporal motion, and fusion decoder produce MANO pose, shape, and metric wrist translation from stereo video.The motion encoder also predicts hand visibility, which gates whether a decoded hand is emitted.Source: §3.4 · p. 4, Figure 3 · p. 4

3.5. Stereo Image Encoder

  • The stereo encoder fuses calibrated views using Plücker-ray embeddings and alternating intra-view and cross-view attention, with random view blanking during training.Training-time heatmap, keypoint, visibility, depth-bin, and triangulation heads guide explicit stereo geometry.Source: §3.5 · p. 4

3.6. Masked Temporal Motion Encoder

  • The motion encoder learns temporal inpainting by reconstructing joints, pose, and root velocity at masked positions and predicting framewise hand visibility.An optical-flow token carries hand and camera motion context, including when image tokens are masked.Source: §3.6 · pp. 4–5

3.7. Query-Gated Fusion Decoder

  • Query-gated fusion lets pose, shape, and wrist queries draw differently on image, motion, and root features, with the wrist leaning on root evidence when image evidence is ambiguous.The wrist is directly regressed in metric coordinates rather than anchored to triangulation; triangulation still contributes through guidance and gating statistics.Source: §3.7 · p. 5

3.8. Training and Supervision

  • Training combines forward-kinematics losses and intermediate encoder supervision with masking and image augmentations that simulate in-the-wild corruption.No oracle signal is provided at test time.Source: §3.8 · p. 5

4. ESTHER3D Benchmark

ESTHER3D pairs a large in-the-wild training set of metric pseudo-labels with a motion-capture test set of true metric ground truth. ESTHER labels the raw collection at scale using an iteratively refined, filtered pipeline, with quality assessed against reprojection measures and motion capture.

4. ESTHER3D Benchmark

  • The benchmark pairs a large in-the-wild training set carrying metric pseudo-labels with a motion-capture test set carrying true metric ground truth.Frames include MANO pose and shape, absolute metric wrist, camera parameters, 2D keypoint projections, and hand boxes; collection spans 194 hours from 121 collectors, 308 venues, 44 scene categories, and roughly 800 tasks.Source: §4 · pp. 5–6

4.1. Labeling with ESTHER and Iterative Enhancement

  • ESTHER refines raw-collection labels through 200 teacher-optimization steps, then applies the teacher’s reprojection and human-review filter.Surviving labels feed back into training.Source: §4.1 · p. 6
  • The teacher reaches 15.9 mm MPJPE and 19.1 mm wrist error against motion capture.In-the-wild labels have a 5.3 px median reprojection residual (2.8 mm median in-plane discrepancy); estimated depth sensitivity is about 21 mm, not a bound on true 3D label error.Source: §4.1 · p. 6

5. Experiments

ESTHER is compared with multiview, challenge-winning, and pretrained monocular architectures under a shared evaluation protocol. It leads on key ESTHER3D and HOT3D results, degrades gracefully when views or frames are missing, and its stereo-guided fallback retains depth information better than a separately trained monocular model.

5.1. Protocol and Metrics

  • All trainable methods share the frozen backbone, data, losses, and budget, isolating architecture; comparisons cover ESTHER3D zero-shot and HOT3D zero-shot and 10%-data fine-tuning.Baselines include stereo-adapted multiview methods, the HOT3D challenge winner, and pretrained monocular models.Source: §5.1 · p. 6, §5.2 · p. 6

5.2. Architecture Comparison

  • On ESTHER3D, ESTHER leads pose (17.7 mm MPJPE) and jitter (3.3 versus 8.5–13.3 m/s2), while HOT3D zero-shot MPJPE is 30.5 mm, beating fine-tuned baselines.After fine-tuning on HOT3D, ESTHER has the lowest MPJPE, PA-MPJPE, wrist error, and jitter among compared methods.Source: §5.2 · pp. 6–7
  • ESTHER’s absolute wrist prediction trades some clean-frame accuracy for robustness to missing views.Its clean-frame wrist error is 21.3 mm versus 20.9–22.5 mm for multiview methods.Source: §5.2 · p. 7

5.3. Robustness

  • With one view missing, ESTHER’s relative pose changes only from 17.7 to 18.0 mm MPJPE, while wrist error rises from 21.3 to 32.5 mm, below baseline wrist errors of 96.8–217 mm.The view-mask model’s one-view HOT3D results are 30.2 mm MPJPE, 12.1 mm PA-MPJPE, 33.9 mm wrist error, and 18.4 m/s2 jitter, lowest among compared methods.Source: Table 2 · p. 6, §5.3 · p. 7
  • Dropping up to 8 of 16 frames leaves ESTHER3D MPJPE and wrist error nearly unchanged, shifting them by only 0.1 mm and 0.2 mm, respectively.The masked-frame objective trains the motion encoder to inpaint missing frames from temporal context; jitter rises overall but not monotonically.Source: §5.3 · p. 7, Table 3 · p. 7

5.4. Why the Monocular Fallback Stays Reliable

  • Across four transformer blocks, the view-mask token attends to surrounding objects, workspace, and arm, unlike stereo and pure monocular models that concentrate on the hand.The paper interprets this scene-context attention as a depth reference supporting inference without disparity.Source: §5.4 · p. 8, Figure 7 · p. 8
  • On held-out data, stereo and view-mask depth correlations are 0.96 and 0.92, with 8 and 11 mm random scatter; pure monocular falls to 0.69 correlation and 32 mm scatter.The view-mask model’s residual depth error is mostly systematic scale compression, while pure monocular predictions regress toward a prior mean plus noise.Source: §5.4 · p. 8, Figure 8 · p. 8

5.5. Ablation

  • The component ablation identifies stereo fusion as the core contributor, with complementary motion cues and auxiliary guidance also supporting performance.Because MPJPE is wrist-relative, the improvement reflects joint configuration and cannot arise solely from more accurate absolute wrist localization.Source: §5.5 · p. 8, Table 4 · p. 8

6. Conclusion

ESTHER combines stereo geometry, temporal reasoning, and metric MANO outputs, with robustness to missing views, dropped frames, and monocular input. The accompanying pipeline builds training labels from stereo evidence and hand priors, then filters them before use.

6. Conclusion

  • ESTHER achieves state-of-the-art accuracy, superior temporal smoothness and generalization, and preserves metric scale under a lost view, dropped frames, or monocular input.Streaming without detection and joint modeling of hands, objects, and bodies remain future work.Source: §6 · p. 8

A. Teacher Pipeline Details

  • The fit uses robust reprojection against filtered stereo keypoints, while triangulated wrist depth provides a soft translation anchor that reprojection can override.The initial seed set uses fused WiLoR estimates; later rounds initialize optimization with ESTHER predictions.Source: A. Teacher Pipeline Details · p. 11
  • Optimization locks shape from 64 high-confidence frames, then refits pose and wrist per frame with shape fixed and applies motion optimization over 16-frame windows.The motion prior regularizes orientation and articulation while allowing genuinely fast motion to remain.Source: A. Teacher Pipeline Details · p. 11
  • The teacher uses stereo only when geometrically reliable and relies on hand models and priors to regularize structure hidden by object or self-occlusion.Its target is articulated hand state, including MANO shape and global wrist translation, rather than the depth of a visible surface.Source: A. Teacher Pipeline Details · pp. 11–12
  • Labels pass a coarse geometric screen, a two-view reprojection check that rejects frames above 15 px residual, and human review for residual failures.The human screen checks issues such as wrong hand identity and implausible articulation.Source: A. Teacher Pipeline Details · p. 12

B. Loss Definitions and Weights

  • Training first geometrically guides the image encoder, then jointly trains the full model with MANO-space losses and masked-position motion supervision.The full-model objective includes joint, mesh, pose, shape, wrist-translation, and articulated-prior terms.Source: B. Loss Definitions and Weights · p. 12

C. Masking and Augmentation Recipe

D. Camera Motion Flow Token

  • The motion encoder receives compensated dense crop flow as an always-present cue, learning to distinguish scene-wide camera motion from hand-localized articulation.Flow is pooled to an 8 × 8 grid; median pooling rejects outliers.Source: D. Camera Motion Flow Token · p. 12

E. Implementation Details

The implementation uses a frozen DINOv3-L backbone and a stereo-temporal model trained in two stages. For HOT3D, fisheye views are rectified to pinhole crops, with trade-offs at the field-of-view periphery.

E. Implementation Details

  • The 354.5M-parameter model uses a frozen DINOv3-L backbone, 16-frame clips, stereo and temporal transformer blocks, and a query-gated fusion decoder.The trainable remainder totals 51.3M parameters; deployment uses the detector only for localization, cropping, and stereo matching.Source: E. Implementation Details · p. 13

F. Camera Setups

  • The head-mounted RGB rig uses synchronized color cameras with a 6.2 cm baseline; HOT3D uses monochrome wide-FOV fisheye cameras that are rectified to pinhole crops.The HOT3D rectification strategy is selected for average accuracy, and ESTHER is nearly invariant to the tested strategy choice.Source: F. Camera Setups · p. 13, Table 1 · p. 13
  • Pinhole rectification supports perspective-trained detectors and the model’s Plucker-ray parameterization, but magnifies peripheral pixels, crops some rim views, and leaves increasing reprojection error at large angles.The grayscale modality and different focal length also contribute to the domain gap addressed by small HOT3D fine-tuning.Source: F. Camera Setups · p. 13

G. ESTHER3D Training Set: In-the-Wild Source Collection

ESTHER3D pairs a large in-the-wild stereo training collection with a motion-capture test set providing metric ground truth. The test protocol covers varied manipulation tasks, and its evaluation chain is checked against independent measurements.

G. ESTHER3D Training Set: In-the-Wild Source Collection

H. ESTHER3D Test Set: Motion Capture Studio

  • The motion-capture test set contains about 10 hours from six subjects performing 50 scripted tasks, with all methods evaluated zero-shot.The calibrated 6.2 cm-baseline stereo rig is synchronized with Vicon for true metric ground truth.Source: H. ESTHER3D Test Set: Motion Capture… · pp. 14–15
  • Its 50 tasks exercise 14 elementary hand actions, ranging from wiping and grasping to bimanual assembly and handovers.The action taxonomy maps each action to concrete manipulation tasks.Source: H. ESTHER3D Test Set: Motion Capture… · p. 15

I. Teacher Pipeline Validation

  • The teacher pipeline is validated without ground-truth access, and camera-frame evaluation uses independently checked transforms and synchronization.The camera mapping has a 10.4 mm residual, while two independent synchronization procedures agree to 10 ms.Source: §I · p. 15

J. Statistical Rigor: Bootstrap Confidence Intervals

  • 17.7 mm MPJPE is ESTHER’s updated point estimate; its jitter 95% interval [3.10, 3.52] lies below the best baseline’s [7.98, 9.17].Wrist error is 21.3 mm, on par with the best multiview methods; bootstrap intervals measure fixed-checkpoint sampling variability, not variation across training seeds.Source: J. Statistical Rigor: Bootstrap… · p. 15, Table 2 · p. 15

K. Robustness to Failure Modes

ESTHER’s metrics remain nearly unchanged under seven photometric and temporal corruptions, including on gloved motion-capture subjects despite training on bare hands.

K. Robustness to Failure Modes

  • The seven corruptions span lighting changes, motion blur, sensor noise, hue shift, and desaturation, while the test subjects’ gloved appearance differs from the bare hands used for training.The authors attribute appearance stability to frozen DINOv3 evidence and heavy image augmentation, and say the temporal encoder absorbs blur as it absorbs dropped frames.Source: K. Robustness to Failure Modes · p. 15
  • 18.2 mm is the worst corrupted-input MPJPE, under sensor noise, within 0.5 mm of the clean 17.7 mm.Hue and illumination shifts leave MPJPE unchanged at 17.7 mm; across corruptions PA-MPJPE stays within 0.2 mm, wrist error within about 2 mm of clean 21.3 mm, and jitter within about 0.4 m/s2.Source: K. Robustness to Failure Modes · p. 15

L. On-Screen Gating and Jitter Protocol

The protocol scores hand pose and motion only on visible frames, defining visibility through joint projections and a predicted visibility gate. The gate also prevents off-screen transitions from inflating jitter.

L. On-Screen Gating and Jitter Protocol

  • MPJPE measures wrist-relative joint error, PA-MPJPE adds Procrustes alignment, and MPJPE-absolute measures joint error without wrist alignment.Frames without a visible hand have no valid target, so gating determines which frames enter each metric.Source: L. On-Screen Gating and Jitter Protocol · pp. 15, 17
  • Jitter is the mean second temporal difference of joint positions across fully on-screen triplets, averaged over 21 joints and scaled by fps^2 at 30 fps.No acceleration is computed across off-screen frames or on/off boundaries.Source: L. On-Screen Gating and Jitter Protocol · p. 17
  • A hand is visible when at least half of its 21 joints project inside either eye; predicted visibility probabilities are thresholded at 0.5.Pose, wrist, and jitter are scored only on frames where the hand is visible.Source: L. On-Screen Gating and Jitter Protocol · p. 17
  • 1.00 precision, recall, and F1 on the full test set; in the blanked-frame stress test, predicted-gated jitter is 3.35 m/s2 versus 4.61 ungated.The on/off/on transition injects a spurious acceleration spike into ungated jitter.Source: L. On-Screen Gating and Jitter Protocol · p. 17

M. HOT3D Robustness to Undistortion Strategy

ESTHER’s predictions are stable across tested HOT3D rectification choices, including focal settings absent from training. The comparison measures each method’s sensitivity to rectification, not absolute accuracy rankings.

M. HOT3D Robustness to Undistortion Strategy

  • The four rectification strategies test within-method stability, not a fair absolute accuracy ranking, because ESTHER and multiview baselines use different training rectifications and weights.ESTHER is trained on offbb120; the multiview baselines use released weights.Source: M. HOT3D Robustness to Undistortion… · p. 17
  • 32.3–32.5 mm MPJPE and 16.3–16.8 m/s2 jitter across offbb focal settings show ESTHER’s stability, including at unseen 90 and 150 px focals.POEM’s wrist error varies from 656 to 450 to 928 mm across the focal sweep; anchor-based methods recover only with the hand-crop strategy.Source: M. HOT3D Robustness to Undistortion… · p. 17

N. Label Quality Analysis

The section assesses in-the-wild label quality through 2D reprojection agreement and geometric sensitivity, while noting that these checks do not bound true 3D label error. It also states the benchmark’s consent and anonymization conditions.

N. Label Quality Analysis

  • The motion-capture test set measures the labeling procedure against true metric ground truth only in the studio; in-the-wild analysis reports 2D agreement and geometric sensitivity.Source: N. Label Quality Analysis · p. 17
  • Table 4 compares robustness across undistortion strategies: ESTHER changes little, including at untrained 90/150 px focals, while multiview baselines vary widely.Source: Table 4 · p. 18
  • 94.6% of hand frames pass the 15 px check, and 94.6% of those passing frames lie below 10 px.The check uses the mean reprojection distance of confident joints across both views; frames above 15 px are rejected.Source: N. Label Quality Analysis · p. 18
  • The approximately 3 mm in-plane discrepancy and 21 mm depth sensitivity do not bound true 3D label error because fitting detectors may share errors with the labels.The 21 mm depth sensitivity assumes independent view errors; opposing view errors yield a separate 30.1 mm median.Source: N. Label Quality Analysis · p. 18

O. Ethics Statement

  • Participants consented to recording and anonymized benchmark release for noncommercial research; recordings contain hands and workspaces, not identifying facial imagery.Source: O. Ethics Statement · p. 18

Related papers

Loading 2609.34817v2…