MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision

Zijie Zhu; Weiren Cai; Yizhou Wang; Zhenjie Yang; Yide Liu; Jiahao Chen; Guanqi He

Source-linked AI summary

MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision

Zijie Zhu, Weiren Cai, Yizhou Wang, Zhenjie Yang, Yide Liu, Jiahao Chen, Guanqi He

arXiv:2609.04958v1Sep 4, 2026cs.CVcs.RO

TL;DR

Egocentric world-space camera and hand supervision is scarce, while existing reconstruction systems separate camera, depth, hand, and refinement stages. MINT jointly predicts camera and bimanual hand states from a shared spatiotemporal representation, trained with scalable EGOPIPELINE supervision, and achieves strong cross-domain performance with zero-shot generalization. Its main scope limitation is inherited monocular scale drift and potential long-horizon drift from the 32-frame training window.

Problem
World-space egocentric supervision is scarce, and existing systems decompose camera and hand reconstruction into separate stages that limit joint modeling.
Method
MINT jointly recovers camera, field of view, bimanual hand state, and hand presence from a unified spatiotemporal representation, using EGOPIPELINE-generated supervision from public video.
Results
MINT achieves state-of-the-art camera trajectory and camera-frame hand-motion performance, with zero-shot generalization to HOT3D and ARCTIC.
Takeaways & Limitations
Large-scale automatically reconstructed motion supervision from public egocentric video can support accurate and generalizable world-space motion reconstruction.
Takeaways & Limitations
MINT remains affected by monocular scale drift inherited from the supervision pipeline, while its 32-frame training window leaves long-horizon drift open.

Abstract

from arXiv · show

Recovering camera and hand motion in world coordinates from egocentric video is a key capability for activity understanding, robot learning, and augmented reality. Existing systems typically decompose this problem into separate stages for camera motion, depth, hand reconstruction, and trajectory refinement, resulting in substantial computational overhead and preventing the joint modeling of camera and hand motion. We introduce MINT (Minting IN-the-Wild Trajectories), the first foundation model that directly produces complete world-space two-hand trajectories from ego-centric RGB video. From a single shared spatiotemporal video representation, MINT jointly predicts the camera trajectory, camera-frame hand states, and per-frame hand presence, and then produces world-space hand motion via explicit coordinate transformations. Training such a model at scale is challenging, since paired world-space camera and hand annotations are scarce. We therefore develop an open-source labeling EGOPIPELINE that converts large collections of public egocentric videos into structured camera-and-hand trajectory supervision. MINT is first pretrained on these large-scale pseudo-labels and then fine-tuned on a small set of high-quality joint annotations. Across public benchmarks, MINT achieves [xxx] improvement in world-space hand trajectory accuracy, [xxx] improvement in camera trajectory estimation, and [xxx] faster end-to-end trajectory generation than the labeling pipeline, while generalizing zero-shot to unseen egocentric datasets. We release the model, training and inference code, labeling pipeline, and a curated 1,021-hour egocentric trajectory dataset.

I. INTRODUCTION

MINT unifies world-space camera and bimanual hand-motion recovery from egocentric RGB video, addressing scarce supervision and the limitations of cascaded reconstruction. EGOPIPELINE supplies scalable motion supervision from ordinary video, while MINT jointly predicts camera, hand, and presence states in one forward pass.

Accurate world-space supervision is scarce because it requires specialized sensing, calibration, synchronization, or costly reconstruction.
Existing systems separately estimate hand, camera, and temporal components, allowing errors to propagate and repeatedly processing shared geometric cues.
MINT jointly decodes camera extrinsics, fields of view, bimanual MANO states, and hand presence from one shared representation without external detection, infilling, optimization, or dense intermediates.
EGOPIPELINE processes 1,729 hours of public egocentric video and combines broad automatically reconstructed data with a small amount of precise geometric supervision.
MINT achieves state-of-the-art camera trajectory and camera-frame hand-motion performance while generalizing zero-shot to HOT3D and ARCTIC.
The released corpus includes reconstructed camera trajectories, bimanual hand state, world-space motion, hand presence, and action descriptions.

II. RELATED WORK

Prior work spans hand reconstruction, world-grounded body motion, camera reconstruction, and controlled 3D datasets. These efforts establish useful components but commonly separate camera estimation from hand or body motion and rely on specialized capture systems.

Large egocentric datasets support learning diverse human actions and visual-language representations, while ViTRA and EgoMimic demonstrate their value for zero-shot prediction and robot co-training.
Hand methods estimate MANO pose and shape from RGB, with newer systems addressing generalization, localization, and interacting two-hand reconstruction.
World-grounded body methods learn camera-aware motion, but they either take camera trajectories as input or recover them separately, unlike MINT’s joint prediction.
EGOPIPELINE instead converts ordinary egocentric video into world-space supervision through hand filtering, camera calibration, depth and pose estimation, hand reconstruction, and post-processing.
Existing 3D supervision often comes from dedicated sensing systems and staged laboratory captures with limited coverage of real-world hand motions and interactions.

C. Feed-Forward 3D Reconstruction

Feed-forward 3D reconstruction has progressed from optimization toward learned estimation of camera and scene geometry. MINT extends this direction to diverse egocentric motion by combining shared geometric representations with scalable video-derived supervision.

DROID-SLAM and MegaSaM combine learned representations with optimization or SLAM to estimate camera poses and scene geometry from dynamic or casual video.
DUSt3R, MASt3R, MonST3R, and CUT3R advance feed-forward reconstruction through point maps, metric correspondence, dynamic-scene modeling, or persistent streaming state.
VGGT, π3, and LingBot-Map estimate camera or scene geometry using cross-view, permutation-equivariant, or spatiotemporal representations for multi-frame and long-stream inputs.
Public egocentric videos provide diverse activities, objects, environments, and interactions that controlled datasets capture less broadly.
EGOPIPELINE estimates intrinsics, metric-scale camera trajectories, and camera-coordinate hand states, then stabilizes outliers and missing intervals for motion supervision.
MINT replaces the pipeline’s heterogeneous stages with joint feed-forward prediction of camera and bimanual motion in a common world coordinate system.

IV. METHOD

MINT adapts a pretrained geometric video encoder for unified egocentric reconstruction. Its bidirectional spatiotemporal features support coupled camera and hand prediction, while overlapping windows enable inference on videos of arbitrary length.

MINT jointly predicts camera and hand motion from a unified visual-geometric representation rather than using separate reconstruction stages.
Training uses two stages: broad egocentric data jointly trains representation and task heads, then high-precision teleoperation data adapts only the camera-extrinsics head.
MINT uses a pretrained GCT encoder because egocentric video interleaves hand motion, contact, camera egomotion, and scene layout that isolated frames cannot fully recover.
The method retains the visual encoder and geometric aggregator, whose frame and global attention propagate spatial and temporal context into shared prediction features.
Global attention is trained on 32-frame clips, so arbitrary-length videos are processed with overlapping sliding windows whose camera trajectories are aligned and chained in SE(3).

C. Prediction Heads

MINT uses shared spatiotemporal geometric features to predict structured hand and camera states, then differentiably maps camera-frame hands into world coordinates. Dedicated observability and iterative refinement address occlusion and temporal discontinuity.

Structured Hand Motion Prediction: MINT adapts pretrained geometric features to decode structured bimanual MANO states, including wrists, joints, and hand shape.Each hand uses four learnable queries corresponding to wrist translation, wrist rotation, joint rotations, and hand shape.
Structured Hand Motion Prediction: Two refinement iterations condition hand predictions on previous outputs to improve temporal consistency during rapid motion and occlusion.The first iteration predicts absolute rotations; later iterations predict bounded local-frame axis-angle increments.
Hand–Camera Geometric Coupling: Camera-frame wrist states are transformed into world-frame hand motion, allowing differentiable world-frame losses to jointly constrain camera and hand predictions.Joint rotations and hand shape remain invariant under the transformation.
Camera State: The camera head predicts translation, rotation, and horizontal and vertical fields of view through iterative refinement and a separate field-of-view branch.Moving field of view outside the feedback path reduces the iterative state from nine to seven dimensions.
Hand Observability: A hand observability head predicts per-hand presence logits so unreliable frames contribute less to hand reconstruction and world-frame losses.Learnable hand queries cross-attend to current-frame patch tokens to produce left- and right-hand presence probabilities.

D. Training Objectives

Training separates broad hand–camera geometry learning from metric camera correction. Stage 1 uses diverse pseudo-supervision and coupled objectives, while Stage 2 uses high-precision camera data to correct scale and absolute pose.

Stage 1: Learning Hand–Camera Coupled Geometry: EGOPIPELINE provides large-scale hand motion and geometric supervision across diverse tasks, scenes, and hand–object interactions.The pipeline reconstructs wrist trajectories, joint motion, and interactions from large-scale egocentric videos.
Stage 2: Camera Trajectory Correction: Monocular depth and SLAM provide useful camera trajectories but can introduce systematic absolute-scale errors, motivating the two-stage design.The first stage learns relative geometric relationships; the second corrects metric scale and absolute pose.
Stage 1: Learning Hand–Camera Coupled Geometry: Stage 1 learns stable hand–camera geometry from diverse Ego4D, EgoDex, and EPIC-KITCHENS clips rather than directly fitting absolutely accurate camera trajectories.The geometric encoder is initialized with pretrained GCT features.
Stage 1: Learning Hand–Camera Coupled Geometry: Stage 1 jointly optimizes camera motion, bimanual MANO, hand presence, and world-frame consistency losses.The objective is L_stage1 = λ_camL_cam + λ_manoL_mano + λ_presL_pres + λ_worldL_world.
Stage 2: Camera Trajectory Correction: Stage 2 uses high-precision camera trajectories to correct residual absolute scale and pose errors while preserving learned hand–camera geometry.Only camera trajectory prediction is optimized after the other components are established or frozen.

A. Setup

Evaluation covers zero-shot camera-frame hand reconstruction and world-frame camera trajectories on HOT3D and ARCTIC. Metrics explicitly account for missed hands, relative and absolute trajectory accuracy, scale deviation, and temporal smoothness.

Datasets: MINT is trained on 1,021 hours of EGOPIPELINE output and evaluated zero-shot on HOT3D and ARCTIC.The training set uses no manual hand annotations or ground-truth camera poses.
Metrics: Coverage-aware hand metrics penalize missed detections instead of evaluating only successfully matched hands.Detection uses FAcc, recall, and F1; camera-frame pose uses MPJPE-p, PA-MPJPE-p, GO-p, and CT-p.
Evaluation Protocol: Table 1 reports camera-frame bimanual reconstruction on ARCTIC and HOT3D under a coverage-aware protocol, with MINT evaluated zero-shot against comparable baselines.ViDiHand is an in-domain reference row, while MINT + UKF changes only inference-time filtering.
Evaluation Protocol: Table 2 evaluates full-length world-frame camera trajectories on 27 HOT3D sequences and 34 ARCTIC validation sequences without fitted scale.This protocol charges scale error and uses arc-length ratio to expose it.
Metrics: World-frame hand motion is evaluated with W-MPJPE, WA-MPJPE, root translation error, and acceleration error over 100-frame segments.Camera trajectories use ATE, ATE%, RPE-T, RPE-R, and arc-length ratio; hand temporal smoothness uses Jitter.

B. Camera-Frame Hands

MINT remains competitive for zero-shot camera-frame hand reconstruction and improves temporal smoothness without test-time optimization. Its explicit presence prediction supports stable detection under egocentric occlusion.

Detection and pose: 0.940 FAcc on HOT3D places MINT near in-domain ViDiHand at 0.948 and above zero-shot WiLoR at 0.827.On HOT3D, MINT also reaches 23.61 mm MPJPE-p and 10.70 mm PA-MPJPE-p.
Temporal smoothness: 79% lower Jitter on both benchmarks after UKF filtering makes MINT smoother while changing MPJPE-p and PA-MPJPE-p by less than 0.1 mm.MINT is already smoother than optimization-based baselines before filtering and remains below in-domain ViDiHand after filtering.

C. Camera Trajectory

MINT's camera trajectory performance reflects a trade-off: it achieves strong relative motion accuracy, while feed-forward windowed inference remains vulnerable to accumulated absolute drift. Two-stage training improves scale calibration and cross-dataset generalization, and mixed-source supervision supports broader transfer.

Camera trajectory: 3.39 mm RPE-T on ARCTIC is best, while 4.69 mm RPE-T on HOT3D is second only to MegaSaM.MINT's HOT3D absolute error is 181.7 mm ATE versus 49.1 mm for DROID-SLAM, because its 32-frame feed-forward windows lack loop closure and global optimization.
Camera trajectory: Stage 2 moves HOT3D arc-length ratio from 0.466 to 1.094 and reduces ATE from 524.7 to 181.7 mm and RPE-T from 8.75 to 4.69 mm.The improvement acts on scale rather than shape and indicates better cross-dataset generalization, despite slightly increasing ARCTIC ATE from 63.7 to 81.9 mm.
Data mixture: The mixed training sources achieve the best result in five of six camera-frame hand reconstruction columns, including 23.61 mm MPJPE-p versus 25.32 mm for the best single source.The mixture is second on Jitter and is selected to generalize across benchmark viewpoints, motions, and intrinsics.
Initialization: Pretrained initialization lowers MPJPE-p, PA-MPJPE-p, and CT-p and reduces Jitter from 18.44 to 11.52 compared with random initialization.Detection is the only metric that slightly improves with random initialization, reaching F1 0.954 versus 0.950.

E. Inference Throughput

MINT substantially reduces inference cost by jointly producing camera and hand outputs in a shared forward pass. It reaches real-time throughput on four GPUs while processing much faster than the full labeling cascade.

Single-GPU throughput: 72.4 ms/frame on one GPU gives MINT 1.45× higher throughput and a 1.51× end-to-end speedup over HaWoR at 105.1 ms/frame.HaWoR's motion-estimation branch alone takes 68.0 ms/frame, close to MINT's joint camera-and-hand inference time.
Multi-GPU throughput: 22.7 ms/frame at four GPUs compares with 83.4 ms/frame for EGOPIPELINE, with utilization of 94.2% versus 43.0%.Windowwise inference scales to 3.27× from one to four GPUs and reaches 44.1 fps, while EGOPIPELINE operates at 0.40× real time.
System design: MINT's single shared forward pass jointly learns world-space camera and bimanual hand motion from large-scale egocentric video.The model is trained using 1,021 hours of automatically extracted motion supervision and generalizes to unseen domains without manual hand annotations or ground-truth camera poses.