PaperlayerView exact revision on arXiv

Source-linked AI summary

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang, Tianyi Wei, Zhiwei Zeng, Siyuan Huang, Zhiqi Shen, Xingang Pan

arXiv:2606.30308v2cs.CV

TL;DR

Egocentric 4D hand-motion reconstruction remains difficult under occlusion and limited hand-pose supervision. ViDiHand adapts a pretrained video diffusion model with hand-overlay rendering and decodes 4D two-hand pose directly from full frames, ranking first on every coverage-aware metric on ARCTIC and HOT3D and eight of nine on held-out HOI4D.

  • Problem

    Existing hand-reconstruction methods struggle with occlusion because they depend on detectors or motion priors learned from scarce, hand-labeled data, limiting reconstruction from egocentric video.

  • Method

    ViDiHand adapts a pretrained video diffusion model with hand-overlay rendering, then uses a dual-branch decoder to recover metric-scale 4D two-hand pose directly from full video frames.

  • Results

    ViDiHand ranks first on every coverage-aware metric on ARCTIC and HOT3D and on eight of nine metrics on held-out HOI4D.

  • Takeaways & Limitations

    The results provide evidence that video diffusion representations can support scalable in-the-wild 4D hand annotation for embodied AI.

  • Takeaways & Limitations

    At 5.5 fps on four A100 GPUs, ViDiHand is currently an offline annotation tool rather than a real-time inference system.

Abstract

from arXiv · show

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

1 Introduction

ViDiHand addresses occlusion-heavy 4D hand-motion reconstruction by adapting pretrained video-diffusion representations with hand-overlay rendering. It achieves state-of-the-art performance across challenging benchmarks without motion infilling or test-time optimization.

  • Motivation: Egocentric 4D hand-motion reconstruction is important for embodied AI but remains difficult because real-world activity contains pervasive occlusion.Egocentric video offers a scalable source for robot learning, with hand and wrist trajectories providing supervision for imitation and policy learning.
  • Limitations: Image-based methods fail when upstream detectors miss heavily occluded hands, while video-based methods rely on scarce hand-pose supervision.These limitations restrict modeling of interaction-rich motion and occlusion-heavy video.
  • Method: ViDiHand is the first method to use a pretrained video diffusion model for 4D two-hand reconstruction from egocentric video.It adapts the model through hand-overlay rendering, specializing features for hands while preserving world priors, then decodes hand motion from internal representations.
  • Results: ViDiHand operates directly on full frames without a detector, motion infiller, or test-time optimization, while producing smoother motion and near-perfect hand detection on heavily occluded ARCTIC sequences.It also leads on most metrics on HOT3D and cross-dataset HOI4D.
  • Results: 27× detection error reduction, at least 4× prediction-jitter reduction, up to 4× lower 2D end-point error, and leading 3D pose accuracy establish state-of-the-art performance.These results hold across ARCTIC, HOT3D, and HOI4D under the coverage-aware protocol for 2D end-point error.

2 Related Work

Prior monocular hand reconstruction methods recover 3D hand pose and shape from RGB input, while video diffusion models have progressed toward large-scale generation of temporally and geometrically coherent video. These developments motivate using diffusion representations for hand-motion reconstruction.

  • Monocular hand reconstruction: Monocular hand reconstruction recovers 3D hand pose and shape from RGB input, commonly parameterized through MANO.Image-based methods estimate hand parameters from a single frame.
  • Monocular hand reconstruction: HaMeR uses a ViT-based regressor trained on heterogeneous 3D and 2D-keypoint datasets to improve generalization to in-the-wild data.
  • Video diffusion models and diffusion features: Video diffusion models evolved from pixel-space VDM and latent-space SVD to large-scale text-to-video transformers such as CogVideoX and the Wan series.These models generate temporally and geometrically coherent video at billion-parameter, internet scale.

3 Method

ViDiHand adapts a pretrained video diffusion model for 4D two-hand reconstruction in two stages: hand-overlay finetuning followed by dual-branch decoding. The decoder combines holistic hand-token reasoning with local joint localization and a differentiable geometric translation solve.

  • Hand-overlay adaptation: ViDiHand finetunes only the VACE branch of Wan2.1-VACE with hand-overlay rendering while freezing the base DiT and preserving scene, object, and interaction priors.The overlays include fully occluded frames and use flow-matching supervision without MANO-parameter supervision.
  • Feature extraction: Features are extracted from the mid-block activation at layer L⋆=15 and denoising step τk≈0.7, then decoded without backpropagation through the diffusion backbone.The selected activation balances spatial resolution for joint localization with abstraction for articulated pose.
  • Dual-branch decoding: The dual-branch decoder uses hand-token slots for holistic articulated MANO pose and joint heatmaps for local 2D coordinates, with mutual cross-attention exchanging complementary evidence.Fixed query slots provide hand identity without a handedness classifier, while spatial-token anchoring matches the local nature of image-space joints.
  • Geometric projection: The mixed-projection head regresses depth and solves in-plane translation (tx, ty) in closed form from predicted joints, anchors, and camera intrinsics.The differentiable weighted least-squares solve jointly optimizes the heatmap, offset, MANO, and depth components while avoiding root-translation/root-pose ambiguity.
  • Training objective: Decoder training combines five losses: Ldec = LMANO + Lcam + Limg + Lvis + Ltemp.These terms supervise MANO, camera-frame geometry, image consistency, visibility, and temporal behavior within the coupled MANO–camera system.

4 Experiments

ViDiHand is evaluated on three complementary egocentric hand benchmarks using coverage-aware metrics that penalize missed hands. It ranks first across nearly all metrics, with gains in detection robustness, pose accuracy, temporal smoothness, and challenging occlusion cases.

  • Datasets: The evaluation covers ARCTIC, HOT3D, and held-out HOI4D, which stress occlusion, distortion, motion blur, lighting, rapid motion, and out-of-distribution generalization.ARCTIC and HOT3D are in-distribution, while HOI4D is held out from explicit supervised training for all methods.
  • Evaluation protocol: Coverage-aware metrics charge false negatives with deterministic placeholder errors, preventing methods from benefiting by skipping difficult frames.The protocol supplements detection metrics with coverage-aware pose, orientation, position, and temporal measures.
  • Quantitative results: 26 of 27 metrics rank ViDiHand first; ARCTIC FAcc reaches 0.997 versus 0.919 for WiLoR, reducing frame-error rate 27× from 8.1% to 0.3%.FAcc requires every on-screen hand to be correctly recovered, making it especially sensitive to missed hands.
  • Quantitative results: 21.5 mm is ViDiHand’s HOT3D MPJPE-p, improving from the best baseline’s 31.0 mm; ARCTIC EPE-p falls from 50.5 px for WildHands to 12.4 px.On co-predicted hands, ViDiHand still outperforms most baselines across the majority of metrics.
  • Quantitative results: ARCTIC jitter drops to 3.18, 4× below Dyn-HaMR’s 12.8, while ViDiHand uses neither a motion prior nor test-time optimization; on held-out HOI4D it ranks first on eight of nine metrics.The reported smoothness reflects backbone coherence plus a lightweight training-time regularizer rather than inference-time engineering.
  • Ablations: Mid-block features and τk≈0.7 perform best, pretrained video backbones outperform alternatives, and removing decoder components reduces FAcc from 0.9979 to 0.9767–0.9829.L15 is the best layer on every metric, while heatmap localization and mixed projection drive image-space placement.

5 Conclusion

ViDiHand reconstructs 4D two-hand motion using a pretrained video diffusion prior specialized by hand-overlay training and decoded into articulated pose and image-space coordinates. It achieves leading coverage-aware performance while remaining an offline tool, motivating distillation, weaker supervision, and broader interaction settings.

  • Contributions: ViDiHand recovers 4D two-hand motion from egocentric video using a pretrained video diffusion prior, hand-overlay specialization, and a dual-branch decoder.Camera translation is anchored by a closed-form mixed-projection solve.
  • Results: ViDiHand ranks first on every coverage-aware metric on ARCTIC and HOT3D and on eight of nine on held-out HOI4D.It pairs near-complete detection coverage with competitive per-hand accuracy without inference-time smoothing.
  • Limitations and future work: At 5.5 fps on 4 A100 GPUs, ViDiHand is currently an offline annotation tool.Distillation and few-step or autoregressive generators are identified as direct paths toward real-time inference.
  • Limitations and future work: Joint-only annotations already drive the Stage-1a overlay pretext, so full MANO supervision is not essential.Future scaling directions include weaker labels, image datasets, additional camera viewpoints, and interactions beyond hand–object interaction.

Supplementary Material

The supplementary material first formalizes the evaluation protocol and metrics, then extends the main-paper results with pairwise pose comparisons, per-side detection F1 breakdowns, and additional qualitative comparisons.

  • Evaluation protocol: The supplement begins by deriving the coverage-aware evaluation protocol and closed-form definitions for every evaluation metric.These materials appear in §A before the subsequent results.
  • Additional evaluations: It adds pairwise pose comparisons on co-predicted hands across all three benchmarks.This comparison extends the main-paper evaluation along one axis.
  • Additional evaluations: It also provides a per-side breakdown of detection F1 and thirteen additional qualitative comparison figures.These extensions are presented in §§C, B, and D.

A Evaluation Protocol and Metric Definitions

The evaluation defines nine metrics across four categories, then aligns predictions with ground truth and applies a coverage-aware protocol that combines detection coverage with pose accuracy. All pose metrics use this protocol except Jitter, whose second-order finite difference requires three contiguous tracked observations.

  • Evaluation Protocol and Metric Definitions: Nine metrics are organized into four categories, with definitions, prediction–ground-truth alignment, and coverage-aware evaluation described in sequence.The coverage-aware protocol is introduced in §A.3 after metric definitions (§A.1) and alignment (§A.2).
  • Evaluation Protocol and Metric Definitions: The coverage-aware protocol folds false negatives into every pose metric, so detection coverage and pose accuracy are represented by one number.Metrics computed under this protocol carry the “-p” subscript, indicating that every present ground-truth hand contributes.
  • Evaluation Protocol and Metric Definitions: All pose metrics use the coverage-aware protocol, while Jitter opts out because its second-order finite difference requires three contiguous tracked observations.Detection metrics are higher-better; the remaining metrics are lower-better.

A.1 Metric Definitions · A.2 Prediction–Ground-Truth Alignment

The paper evaluates hand reconstruction with detection, coverage-aware 3D pose, absolute orientation and position, reprojection, and temporal-smoothness metrics. Predictions are aligned to ground truth using permissive handedness-constrained IoU matching, visibility thresholds, and off-screen exclusion.

  • A.1 Metric Definitions: The evaluation uses a consistent 21-joint ordering, with joint 0 as the wrist, across loss terms and metrics.Predicted joints are camera-frame positions, paired with corresponding ground-truth joints.
  • A.1 Metric Definitions: Detection metrics assess left/right hand detection independently of pose accuracy using corpus-level counts across test frames and segments.They include strict frame accuracy, recall, and F1, which balances false positives and false negatives.
  • A.1 Metric Definitions: MPJPE-p and PA-MPJPE-p measure root-relative 3D joint accuracy, while coverage-aware aggregation penalizes missed hands with a canonical placeholder.MPJPE-p removes absolute translation; PA-MPJPE-p additionally removes global orientation, translation, and scale on matched positives, while retaining a raw missed-hand cost.
  • A.1 Metric Definitions: Absolute metrics evaluate global wrist orientation, camera-frame translation, and 2D reprojection rather than root-relative articulation.GO-p reports geodesic rotation error, CT-p measures 3D wrist placement, and EPE-p aggregates projected joint errors using on-screen masks.
  • A.1 Metric Definitions: Jitter measures mean second-order finite-difference magnitude in mm/frame2 over contiguous true-positive runs of at least three frames.False negatives break runs, and jitter is computed without coverage-aware placeholders because temporal differences require contiguous tracked identities.
  • A.2 Prediction–Ground-Truth Alignment: Each frame’s predicted and ground-truth MANO meshes are matched greedily by descending bounding-box IoU after 10% ground-truth-box dilation.A match requires equal handedness and IoU > τIoU=0.1, a deliberately permissive threshold for ambiguous near-misses.
  • A.2 Prediction–Ground-Truth Alignment: A matched prediction is a true positive when its hand-presence probability exceeds 0.5, with confidence interpreted according to each baseline’s output type.The method thresholds σ(ˆe) > 0.5; explicit-confidence baselines use reported probabilities, while tracker pipelines treat every emitted prediction as positive.
  • A.2 Prediction–Ground-Truth Alignment: Ground-truth hands are off-screen when none of their 21 MANO joints projects inside the image with z > 0.01 m, and such hands are excluded from evaluation.The joint-based criterion aligns with MPJPE and EPE, avoiding classification of a hand as visible from only a mesh fingertip sliver.

A.3 Coverage-Aware Evaluation Protocol

The coverage-aware protocol evaluates every ground-truth hand, including missed detections, by assigning false negatives a canonical MANO placeholder error. This produces metrics that jointly reflect detection coverage and pose accuracy rather than evaluating only true positives.

  • Motivation: Standard pose metrics evaluate only true-positive detections, potentially rewarding conservative methods that skip ambiguous frames.A method detecting 85% of hands with excellent per-prediction accuracy can appear superior to one detecting 97% with slightly lower accuracy.
  • Placeholder construction: For each false negative, the protocol substitutes a canonical MANO placeholder: an identity-rotation, zero-pose, mean-shape hand at the camera origin.
  • Placeholder error: The canonical placeholder yields about 132 mm per-sample MPJPE on ARCTIC after wrist anchoring, reflecting joint displacement from rest-pose to typical articulation.Because MPJPE is root-relative, absolute depth cancels out.
  • Coverage-aware metric: The resulting metric jointly captures detection coverage and pose accuracy, imposing a substantial deterministic cost for missed hands while penalizing inaccurate predictions.Placeholder error is computed per sample from the missed hand’s actual pose, orientation, and position, rather than treated as a fixed constant.

B Per-Side Hand Detection Analysis

Per-side detection analysis isolates left–right confusion using left, right, and total F1 across three benchmarks. ViDiHand remains highly consistent, while baseline asymmetry is most severe on HOI4D due to spurious predictions for absent hands.

  • HOI4D asymmetry: On HOI4D, five baselines reach left-hand F1 values between 0.385 and 0.473, the most severe asymmetry across datasets.Most clips show single-handed manipulation, causing these pipelines to emit spurious predictions on the absent side.
  • ViDiHand performance: ViDiHand maintains per-side F1 at or above 0.981 on every dataset.This result indicates consistent left–right hand detection across the evaluated benchmarks.
  • Per-side detection metrics: Table 7 evaluates left-hand, right-hand, and combined detection F1 for eight baselines and ViDiHand across three benchmarks.Higher F1 is better, and the analysis treats asymmetric per-side F1 as the measure of left–right confusion.

C Pairwise Comparison on Co-Predicted Hands · D Additional Qualitative Comparisons

The co-predicted-hands comparisons use a protocol favorable to baselines by excluding missed hands, yet ViDiHand shows strong qualitative robustness across occlusion, truncation, peripheral views, and hand-object interactions. Additional comparisons cover ARCTIC, HOT3D, and in-the-wild clips using projected joints, mesh overlays, and novel-view renderings.

  • C Pairwise Comparison on Co-Predicted Hands: Tables 8–10 report baseline / ViDiHand metrics only where both methods predict the same hand side and the ground-truth hand is visible.This conventional protocol aggregates pose over co-predicted hands and lets missed hands contribute nothing.
  • C Pairwise Comparison on Co-Predicted Hands: The protocol favors baselines by excluding difficult missed hands, while baseline wins on in-distribution data concentrate on Procrustes-aligned MPJPE.Under this alignment, strong crop-based and SLAM-refined baselines can match or surpass ViDiHand on detected hands.
  • C Pairwise Comparison on Co-Predicted Hands: On held-out HOI4D, baseline wins broaden across Procrustes-aligned MPJPE, 2D reprojection, and camera translation.ViDiHand is trained on ARCTIC and HOT3D without explicit HOI4D supervision, although incidental exposure from its pretrained corpus cannot be ruled out.
  • D Additional Qualitative Comparisons: Qualitative figures compare eight baselines with ViDiHand through projected 2D joints, reprojected MANO overlays, and three novel-viewpoint 3D renderings.The comparisons span ARCTIC, HOT3D, and in-the-wild clips; ground-truth MANO is omitted for in-the-wild examples.
  • D Additional Qualitative Comparisons: Across ARCTIC examples, ViDiHand alone consistently recovers both hands and plausible articulation when objects, frame boundaries, or occlusion hide most of the hands.Baselines frequently miss one or both hands, distort articulation, or place occluded hands at incorrect depth.
  • D Additional Qualitative Comparisons: Across HOT3D examples, ViDiHand preserves hand position, orientation, depth, and articulation in dim, fisheye, peripheral, blurred, and heavily occluded scenes.Baselines commonly miss hands, collapse meshes, or predict incorrect position, depth, and articulation.
  • D Additional Qualitative Comparisons: In-the-wild examples show ViDiHand recovering plausible bimanual or single-hand poses under cloth occlusion, shelf-edge occlusion, uneven lighting, and severe depth or rotation errors.Baselines miss hands, duplicate detections, produce blobs or phantom hands, or misestimate depth, rotation, and articulation.

E Implementation Details

The implementation specifies the full pipeline through its data, decoder, losses, and two-stage training procedure. Its video backbone is Wan2.1-VACE, whose VACE branch conditions on the egocentric input video.

  • The section consolidates the data, decoder architecture, losses, and two-stage training procedure defining the full pipeline.
  • Wan2.1-VACE is a 1.3-billion-parameter video diffusion transformer with a VACE branch that injects the egocentric input video as conditioning.
  • L15 denotes the 15th of 30 transformer blocks, while τ_k denotes the normalized denoising step from pure noise at τ=0 to fully denoised at τ=1.

E.1 Datasets and Cameras … E.5 Controlled Fitting Study: Capacity of the Feature Slice

The appendix specifies the datasets, camera-aware decoder, losses, and staged training used by ViDiHand, then tests whether the selected mid-denoise feature slice can support MANO decoding. It also documents dataset exposure and held-out evaluation conditions for interpreting the experiments.

  • E.1 Datasets and Cameras: All clips use consecutive 81-frame segments, compressed to 21 latent frames, with dataset-specific spatial patch grids after resizing to height 480.The grids are 30×42 for ARCTIC, 30×30 for HOT3D, and 30×54 for HOI4D; HOI4D is padded from 854 to 864 pixels before patchification.
  • E.1 Datasets and Cameras: HOI4D is a held-out 15-fps test set covering 800+ object instances across 16 categories, and no method, including ViDiHand, trains on it.Its mostly single-handed clips require reporting “no hand on screen” for long consecutive intervals.
  • E.2 Decoder Architecture; E.2.1 Ray-Space Positional Encoding: The 37-million-parameter decoder reads fixed Stage-1b DiT activations and combines per-hand token and joint-heatmap branches through mutual cross-attention to predict MANO parameters.The hand-token branch uses ray-space encoding, while the heatmap branch produces 21 joint estimates and visual descriptors per hand.
  • E.2.1 Ray-Space Positional Encoding: Ray-space positional encoding computes camera-ray directions from pinhole intrinsics and adds learned camera-aware corrections residually to spatial positional encoding.Its contribution is zero-initialized, so the decoder initially uses spatial encoding alone and learns ray-space corrections gradually.
  • E.2.2 Mixed-Projection Camera-Translation Head: The Mixed-Projection Head regresses log-depth but solves in-plane translation analytically from refined heatmap anchors using independent scalar least-squares problems.All 21 joints participate when valid, with per-frame and per-hand masking adapting the effective system size.
  • E.4 Two-Stage Training Pipeline; E.4.1 Stage 1a: Joint-Overlay Pretraining on EgoDex; E.4.2 Stage 1b: MANO Mesh-Overlay Finetuning; E.4.3 Stage 2: MANO Decoder Training: Training adapts VACE with EgoDex joint-overlay pretraining, ARCTIC/HOT3D mesh-overlay finetuning, and decoder training on cached Stage-1b features.Stage 1a runs 25 k steps, Stage 1b runs 10 k steps with sampling weights (0.283, 0.717), and Stage 2 trains for 30 k steps.
  • E.5 Controlled Fitting Study: Capacity of the Feature Slice: A controlled single-segment ARCTIC fitting study tests whether the frozen default L15, τk≈0.7 feature slice contains sufficient information for accurate, temporally consistent MANO decoding.The decoder is fit with the full decoder loss for 3 k steps before evaluating MPJPE-p.

F Loss-Term Ablation on ARCTIC

The ARCTIC loss-term ablation removes one decoder loss at a time while holding the training protocol and backbone fixed. Every removal worsens MPJPE-p, while acceleration smoothness specifically regularizes temporal jitter.

  • Ablation protocol: Each ablation removes one loss term from the full decoder objective while holding all other terms, the training protocol, and the backbone fixed.Stage-2 decoder training uses ARCTIC alone, while the Stage-1b VACE backbone remains trained on ARCTIC + HOT3D.
  • Loss-term effects: Removing any single loss term increases MPJPE-p; shape consistency causes the smallest increase at 0.12 mm, while geodesic rotation causes the largest at 0.84 mm.The comparison is relative to the full objective.
  • Loss-term effects: Removing acceleration smoothness raises Jitter from 3.42 to 3.88 mm/frame2, making it the only loss that materially changes this metric.This result identifies acceleration smoothness as a temporal regularizer.
Loading 2606.30308v2…