PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery

Chen Xu, Yunqi Li, Binbin Huang, Brent Yi, Shenghua Gao, Yi Ma

arXiv:2610.10512v1cs.CV

TL;DR

Occlusions and incomplete visual evidence make faithful hand-motion recovery from video challenging, and 2D trajectory dynamics remain underexplored. JoHan jointly generates aligned 2D and 3D motion from video, with experiments reporting accurate reconstruction and smoother motion than prior methods.

  • Problem

    Occlusion reduces visual evidence, and the continuity and dynamics of 2D hand trajectories—which can affect 3D recovery through geometric projection—remain comparatively underexplored.

    Source: §1 · p. 1

  • Method

    JoHan jointly generates aligned local 2D and 3D hand motion directly from video without per-frame pose estimates, using evolving 2D predictions to guide 3D generation and correspondences for global camera-space recovery.

    Source: §1 · p. 2, §4 · p. 9

  • Results

    JoHan achieves 32.8 mm MPJPE on HO3Dv3, the lowest among compared methods, and on HOT3D its 2.69 m/s2 acceleration error improves over HaWoR by 62.3%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results support accurate local and camera-space reconstruction with smoother motion across static- and moving-camera settings.

    Source: §1 · p. 2

    On HOT3D, W-MPJPE and WA-MPJPE remain slightly behind state-of-the-art results because the method focuses on local motion modeling rather than explicitly learning world-coordinate hand motion.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, we propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions. Trained from scratch, our model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence. The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the following generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency. Their learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera. Extensive experiments on challenging benchmarks demonstrate significantly improved accuracy and speed in local hand-pose and camera-space reconstruction. Notably, our method captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.

1 INTRODUCTION

Occlusion and incomplete visual evidence make hand-motion recovery difficult, while existing methods leave the temporal dynamics of 2D trajectories underexplored. JoHan jointly generates aligned 2D and 3D motion from video, and experiments report accurate reconstruction and smoother motion.

1 INTRODUCTION

  • Existing generative methods use per-frame estimates as motion conditions or optimization initializations, so errors can affect recovery, particularly under persistent occlusion (Xu et al., 2026).Earlier deterministic methods produce a single pose or trajectory and cannot explicitly represent multiple plausible solutions under ambiguity.Source: §1 · p. 1, RELATED WORK · p. 3
  • Although 2D trajectories provide spatial and temporal evidence for 3D recovery, their continuity and dynamics remain less explored; their errors can compromise 3D motion through geometric projection.Prior work uses 2D image-space constraints and correspondences, but joint modeling of 2D trajectory dynamics and 3D motion remains comparatively underexplored.Source: §1 · p. 1, RELATED WORK · p. 3
  • Its evolving 2D trajectories guide 3D generation through correspondence, combining pixel-grounded visual constraints with 3D motion dynamics to improve recovery over 3D generation alone.The approach uses a unified flow-matching process and cross-modal supervision, then estimates global orientation and translation using a PnP solver.Source: §1 · pp. 1–2
  • On HO3Dv3, JoHan achieves 32.8 mm MPJPE, the lowest among compared methods; on HOT3D, its 2.69 m/s2 acceleration error improves over HaWoR by 62.3%.On HOT3D, JoHan also improves over HaWoR by 21.5% in PA-MPJPE; the reported values are 4.75 mm for JoHan.Source: §1 · p. 2
  • JoHan jointly generates aligned local 2D and 3D pose sequences from video without relying on pretrained per-frame pose estimators, and uses their correspondences to recover global camera-space hand pose.The learned sequences model temporal dynamics within a shared motion-generation process.Source: §1 · p. 2

2 METHODOLOGY

JoHan generates aligned local 2D and 3D hand motion directly from video, using evolving 2D predictions to guide 3D generation. It then aligns the generated local poses with restored 2D coordinates to recover the hand’s global camera-space motion.

2.1 PROBLEM FORMULATION

  • The model generates aligned local 2D and 3D sequences conditioned on video, hand masks, and hand side, separating local motion from global camera placement.The 2D representation removes image-plane location and scale; local 3D motion excludes global orientation and translation.Source: §2.1 · p. 4

2.2 GENERATION-BASED JOINT 2D–3D MOTION RECONSTRUCTION

  • Shared-timeline flow matching generates paired local 2D and 3D motion, with the evolving 2D prediction encoded as guidance for the 3D branch at each step.The guidance pairs each MANO joint with its corresponding 2D joint feature, weighted by predicted visibility.Source: §2.2 · pp. 5–6
  • Video-conditioned generators use full-frame and hand-mask features, while joint training supervises clean-motion predictions and reprojection consistency.The masks locate the hand, while full-frame features retain visual context; mask conditions may be dropped during training.Source: §2.2 · pp. 5–6

2.3 GLOBAL MOTION RECOVERY

  • After restoring generated 2D poses to image coordinates, a PnP solver aligns them with local MANO joints to estimate global orientation and camera-space wrist position.The recovered global parameters transform the generated local poses into full camera-space hand motion.Source: §2.3 · p. 6

3 EXPERIMENTS

JoHan is evaluated for local pose and mesh accuracy, camera-space and world-space recovery, motion smoothness, and hand-recovery efficiency. Across HO3D and HOT3D, results support accurate local reconstruction and smoother motion, while ablations examine supervision, inputs, and 2D–3D generation choices.

3.1 EXPERIMENTAL SETUP

  • JoHan is trained on six datasets and evaluated for local pose accuracy on HO3Dv2/v3 and motion continuity and smoothness on HOT3D.The reported measures include aligned joint and vertex errors, camera-space MPJPE, image-plane Err2D, world-space errors, and acceleration error.Source: §3.1 · p. 7

3.2 QUANTITATIVE RESULTS

  • 7.0 mm PA-MPJPE, 6.9 mm PA-MPVPE, and 4.19-pixel Err2D are the lowest among compared methods on HO3Dv3; HO3Dv2 results are also best on these local metrics.After global recovery, HO3Dv3 MPJPE is 32.8 mm, versus 53.5 mm for WiLoR and 54.7 mm for HaMeR.Source: §3.2 · pp. 7–8
  • 4.75 mm PA-MPJPE and 2.69 m/s2 AccEr accompany the best local pose accuracy on HOT3D, although W-MPJPE and WA-MPJPE remain slightly behind state of the art.The authors attribute the world-space gap to recovering trajectories by transforming camera-relative motion with estimated camera motion rather than learning world-space motion directly.Source: Table 2 · p. 8, §3.2 · p. 8
  • 2.03 s per 128-frame clip is the lowest total hand-recovery time, 27.0% of WiLoR’s 7.52 s, with 351M versus 630M parameters.The comparison excludes shared camera processing and uses one NVIDIA H200 GPU.Source: §3.2 · p. 8, Table 3 · p. 8

3.3 ABLATIONS

  • Reprojection supervision reduces PA-MPJPE from 8.2 to 7.8 mm and Err2D from 5.96 to 5.21 pixels in the HO3Dv3 ablation.The ablation uses DINOv3 ViT-B/16; the 3D joint loss slightly improves both outputs, while reprojection supervision has the larger effect.Source: §3.3 · pp. 8–9, Table 4 · p. 9
  • Detected hand crops raise PA-MPJPE from 7.8 to 8.9 mm and Err2D from 5.21 to 15.68 pixels, while removing mask conditions yields 8.0 mm PA-MPJPE and about 7.8-pixel Err2D.The authors report that crop resizing can disrupt visual continuity; masks offer modest local 3D and larger 2D benefits, while full frames and hand-side labels still support accurate local 3D poses.Source: §3.3 · p. 9
  • Joint generation improves both modalities: PA-MPJPE falls from 8.9 to 7.8 mm versus a 3D-only generator, and Err2D from 5.64 to 5.21 pixels versus a 2D-only generator.Separate training with ground-truth 2D conditioning yields 12.89 mm PA-MPJPE; noisy teacher forcing improves this to 9.49 mm, still above JoHan’s 7.8 mm.Source: §3.3 · p. 9

4 CONCLUSION

JoHan jointly generates local 2D and 3D hand motion directly from monocular video, then uses geometric alignment to recover camera coordinates. Experiments report accurate HO3D reconstruction and smoother HOT3D motion, with ablations supporting joint generation and reprojection supervision.

4 CONCLUSION

  • JoHan jointly generates aligned local 2D and 3D motion from monocular video without relying on per-frame pose estimates.The evolving 2D prediction guides 3D generation, and reprojection supervision promotes geometric consistency.Source: §4 · p. 9
  • Geometric alignment transforms generated local 3D motion into camera coordinates using recovered global orientation and translation.Source: §4 · p. 9
  • HO3D experiments show accurate local poses and camera-relative hand recovery, while HOT3D results show smoother motion with accurate local poses and high speed.Ablations report benefits from joint generation and reprojection supervision, and accurate local 3D reconstruction without detection cues.Source: §4 · p. 9

AI USAGE STATEMENT

The authors report using generative AI for language, figure, implementation, and mathematical-formulation assistance. They state that research ideas and experimental designs were theirs, results came from supervised experiments, and AI was not used to fabricate data or measurements.

AI USAGE STATEMENT

  • Generative AI assisted with translation and language editing, figure preparation and refinement, selected code implementation and execution, and clarification of mathematical formulation and results.Source: AI USAGE STATEMENT · p. 10
  • The authors state that they conceptualized the research ideas and experimental designs, with AI assistance implementing and running parts of experiments.Source: AI USAGE STATEMENT · p. 10
  • The authors report that experiments produced the results under their supervision and that AI tools did not fabricate data or measurements.Source: AI USAGE STATEMENT · p. 10

A APPENDIX

The appendix includes qualitative comparisons of hand reconstructions on HO3D and occluded HOT3D frames, plus world-space motion comparisons on HOT3D. It also provides a conditioned-generator architecture figure.

A APPENDIX

  • Figure 3 compares JoHan, HaMeR, and WiLoR on selected HO3D frames using crop projections and two camera-space mesh views overlaid with ground truth.Ground truth is green; the methods are blue, orange, and purple, respectively.Source: Figure 3 · p. 13
  • Figure 4 shows six selected HOT3D frames with hand–object occlusion, comparing the three methods through crop projections and camera-space mesh views.Its layout and colors follow Figure 3.Source: Figure 4 · p. 14
  • Figure 5 compares four HOT3D world-space examples from two sequences, with aligned meshes and trajectories projected onto the first two ground-truth principal axes.Grid spacing is 10 cm; curves show per-frame W-MPJPE and markers identify displayed frames.Source: Figure 5 · p. 15
  • Figure 6 depicts conditioned generators; the 3D generator uses evolving 2D guidance before motion tokenization, and both generators apply visual cross-attention followed by spatial and temporal self-attention.Timestep and hand-side conditioning are indicated by gold dashed arrows.Source: Figure 6 · p. 16, A APPENDIX · p. 16

A.1 IMPLEMENTATION DETAILS

JoHan uses jointly trained 2D and 3D motion generators conditioned on full-frame video, with 2D predictions guiding 3D generation. Training and inference use paired motion representations and a ten-step joint flow schedule.

  • Training used 32 GPUs for 300,000 iterations with an effective batch size of 256 clips; the generators were trained from scratch and the pretrained DINOv3 ViT-L/16 backbone was fine-tuned jointly.Hand-mask conditions were independently dropped per frame with probability 0.3, while RGB video remained available.Source: §A.1 · p. 16
  • At inference, both motion states start from independent Gaussian noise and undergo 10 joint flow steps, recomputing 2D before 3D at each step; framewise PnP recovers global orientation and translation.Box trajectories are Kalman-filtered.Source: §A.1 · pp. 16–17
  • The independent 2D and 3D generators use 384-dimensional tokens and four and six spatiotemporal blocks, respectively, with visual cross-attention followed by spatial and temporal self-attention.The 2D branch predicts coordinates and visibility; the 3D branch predicts MANO pose and shape parameters.Source: §A.1 · p. 16, Table 5 · p. 17
  • DINOv3 image features are fused with pooled hand-mask features, while the 3D branch receives visibility-weighted 2D joint guidance, including wrist and fingertip guidance for its shape token.The guidance is generated by the 2D branch during both training and inference.Source: §A.1 · pp. 16–17

A.2 QUALITATIVE RESULTS

Qualitative examples show JoHan recovering hand poses and image placement under hand–object occlusion, as well as smoother world-space paths than HaWoR in the shown HOT3D sequences.

  • JoHan’s examples show detailed hand poses under hand–object occlusion while preserving image placement relative to the observed hand and manipulated object.The mesh views compare finger poses and 3D alignment with ground truth against HaMeR and WiLoR.Source: §A.2 · p. 18
  • In four HOT3D examples, JoHan follows smoother world-space paths with fewer abrupt changes than HaWoR, consistent with lower acceleration errors reported in Tables 2 and 7.Per-frame W-MPJPE curves separately show world-space reconstruction accuracy over each sequence.Source: §A.2 · p. 18

A.3 DATA PREPROCESSING

The preprocessing pipeline standardizes data from six hand-motion datasets into a camera-space representation and constructs temporally ordered clips with validity masks. Images and annotations are normalized consistently for training and evaluation.

  • Training combines Inter-Hand2.6M, DexYCB, ARCTIC, HO3D, Re:InterHand, and HOT3D, converting annotations to a common camera-space representation with MANO parameters, 21 joints, intrinsics, hand side, and validity masks.Left- and right-hand tracks are treated as independent sequences, and held-out evaluation sequences are excluded before clip construction.Source: §A.3 · p. 18
  • Full RGB frames are resized to 256 × 256 without hand-centered cropping, with camera intrinsics updated consistently and specified egocentric images undistorted before resizing.The HOT3D Aria stream also follows HaWoR’s image-orientation convention, including a 90° clockwise rotation.Source: §A.3 · p. 18
  • Projected 3D annotations provide 2D joint targets, whose per-axis minima and maxima define local 2D normalization bounds and reprojection supervision.These training bounds differ from detector-derived bounds used for inference denormalization.Source: §A.3 · p. 18
  • Tracks are split at sequence boundaries and annotation discontinuities; clips shorter than 32 valid frames are discarded, while longer tracks use windows of up to 128 frames with stride 16.Retained clips are padded to 128 frames, and temporal validity masks exclude padded frames from supervision.Source: §A.3 · p. 18

A.4 EVALUATION DETAILS

Evaluation uses official HO3D sets and held-out HOT3D Aria recordings, with metrics and evaluation data selected for each benchmark. HOT3D world-space reconstruction uses estimated camera trajectories supplied by HaWoR’s camera-motion module.

  • HO3Dv2 and HO3Dv3 evaluation uses released ground-truth joints and mesh vertices and reports PA-MPJPE, PA-MPVPE, AUCJ, F@5, and F@15.HO3Dv3 also reports MPJPE without Procrustes alignment for camera-space recovery and Err2D against projected ground-truth joints in resized image coordinates.Source: §A.4 · pp. 18–19
  • HOT3D evaluation uses Aria clips with annotated hand poses and camera trajectories; the official test split lacks released ground truth, so evaluation uses 27 held-out official-training recordings.The held-out recordings are excluded from model training.Source: §A.4 · p. 19
  • The main HOT3D evaluation draws on 333 clips from 25 held-out recordings, with 42,990 right-hand frames after validity and shared detection-mask filtering.HaWoR’s camera-motion module supplies estimated camera trajectories only for transforming reconstructed camera-space motion into world coordinates.Source: §A.4 · p. 19

A.5 BOUNDING-BOX REFINEMENT FOR GLOBAL RECOVERY

Detector boxes can differ from the joint-derived bounds used to normalize training poses, introducing scale and shift errors that can affect geometric global recovery. JoHan uses a separately trained bounding-box refiner, which improves image-space, camera-space, and world-space errors on HOT3D.

A.5 BOUNDING-BOX REFINEMENT FOR GLOBAL RECOVERY

  • Detector boxes may differ from joint-derived bounds, so direct denormalization can introduce scale and shift errors that affect PnP-estimated orientation and translation.Source: §A.5 · p. 19
  • A separately trained refiner uses image-region CNN features and box-coordinate embeddings to predict residual scale and center corrections before 2D motion denormalization.It is trained by perturbing ground-truth joint-derived boxes and supervising recovery of their original bounds.Source: §A.5 · p. 19
  • With refinement, Err2D falls to 3.27 px from 5.18 px, camera-space MPJPE to 33.84 mm from 46.05 mm, and W-MPJPE to 55.05 mm from 78.37 mm.Table 6 compares refinement with no refinement on the same 42,990 HOT3D frames; lower values are better.Source: Table 6 · p. 19, §A.5 · p. 19

A.6 HOT3D EVALUATION WITH GROUND-TRUTH BOUNDING BOXES

With ground-truth hand boxes on 100-frame HOT3D segments, JoHan retains lower acceleration error while achieving a lower PA-MPJPE than HaWoR. Its W-MPJPE and WA-MPJPE are comparable but slightly higher.

A.6 HOT3D EVALUATION WITH GROUND-TRUTH BOUNDING BOXES

  • JoHan's PA-MPJPE is 4.70 mm versus HaWoR's 5.47 mm; W-MPJPE is 45.78 versus 42.95 mm and WA-MPJPE is 14.89 versus 13.43 mm.Source: Table 7 · p. 20, §A.6 · p. 20
  • The evaluation uses ground-truth hand boxes and 100-frame segments, with JoHan using HaWoR's camera motion estimation module.Because these segments are shorter than the main HOT3D-Clips comparison, the authors assess methods within Table 7.Source: Table 7 · p. 20, §A.6 · p. 20
  • JoHan's AccEr is 3.34 m/s2 versus HaWoR's 5.13 m/s2 and is also lower than HaMeR's and WiLoR's.The authors interpret the lower acceleration error as evidence that JoHan retains its smoothness advantage without detector-box preprocessing.Source: §A.6 · p. 20

Related papers

Loading 2610.10512v1…