PaperlayerView exact revision on arXiv

Source-linked AI summary

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

Yuxi Wang, Chengkai Jin, Yufei Liu, Wenqi Ouyang, Tianyi Wei, Zhiwei Zeng, Siyuan Huang, Zhiqi Shen, Xingang Pan

arXiv:2606.30308v2cs.CV

TL;DR

Egocentric 4D hand-motion reconstruction struggles with occlusion and limited hand-pose supervision, despite its importance for scalable embodied-AI learning. ViDiHand adapts a pretrained video diffusion model for hand-aware reconstruction and achieves state-of-the-art results across three benchmarks.

  • Problem

    Existing methods struggle with egocentric 4D hand-motion reconstruction because detectors fail under occlusion and temporal priors rely on scarce hand-labeled data.

    Source: §1 · p. 2

  • Method

    ViDiHand adapts a pretrained video diffusion model with hand-overlay rendering and decodes adapted representations into metric-scale two-hand motion from full video frames.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    ViDiHand ranks first on every coverage-aware metric on ARCTIC and HOT3D and on eight of nine metrics on held-out HOI4D.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Video diffusion representations show potential as a foundation for hand reconstruction and scalable in-the-wild 4D hand annotation.

    Source: §1 · p. 3, §5 · p. 9

    At 5.5 fps on 4 A100 GPUs, ViDiHand is currently an offline annotation tool rather than a real-time system.

    Source: §5 · p. 10

Abstract

from arXiv · show

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

1 Introduction

ViDiHand addresses occlusion-heavy 4D two-hand reconstruction by adapting a pretrained video diffusion model with hand-overlay rendering. It achieves state-of-the-art performance across challenging benchmarks while avoiding detectors, motion infilling, and test-time optimization.

Motivation

  • Existing image-based methods fail when upstream detectors miss heavily occluded hands, while video-based methods rely on scarce hand-pose supervision that inadequately captures motion dynamics and interactions.These limitations are especially important for embodied AI, where egocentric video offers scalable supervision and hand or wrist trajectories support imitation and policy learning [12] [9].Source: §1 · p. 2

Contribution

  • ViDiHand is the first method to leverage a pretrained video diffusion model for 4D two-hand motion reconstruction from egocentric video.It decodes hand motion from internal representations containing structured information about hands, manipulated objects, and the scene.Source: §1 · p. 2
  • Hand-overlay rendering adapts the diffusion model toward hand-aware reconstruction while preserving its world priors.The objective edits the hand region while reconstructing the rest of the scene, specializing features for hands without discarding broader scene representations.Source: §1 · p. 2

Results

  • ViDiHand ranks first on nearly every metric, cutting detection error by up to 27×, prediction jitter by at least 4×, and 2D endpoint error by up to 4× while leading 3D pose accuracy across all three benchmarks.On ARCTIC [5], it achieves near-perfect hand detection and smoother motion than prior methods; it also leads on most metrics on HOT3D [1] and HOI4D [15].Source: §1 · pp. 2–3

2 Related Work

Related work spans monocular hand reconstruction from RGB, commonly parameterized with MANO, and video diffusion models whose scale and coherence motivate their use as motion representations. Image-based methods improve in-the-wild generalization, while diffusion research has progressed toward large-scale coherent generation and controllable conditioning.

Monocular hand reconstruction

  • Monocular hand reconstruction recovers 3D hand pose and shape from RGB input, commonly using the MANO parameterization.This line of work includes image-based methods that estimate hand parameters from a single frame.Source: §2 · p. 3
  • HaMeR [18] improves in-the-wild generalization with a ViT-based regressor trained on heterogeneous 3D and 2D-keypoint datasets.Source: §2 · p. 3

Video diffusion models and diffusion features

  • Video diffusion models progressed from pixel-space VDM and latent-space SVD [2] to large-scale text-to-video transformers such as CogVideoX [30] and Wan [26].The passage characterizes these newer models as generating temporally and geometrically coherent video at billion-parameter, internet scale; VACE [11] further augments Wan2.1 with a unified conditioning path.Source: §2 · p. 3

3 Method

ViDiHand adapts a pretrained video diffusion model for two-hand 4D reconstruction through hand-overlay rendering, then decodes articulated pose and camera translation with complementary hand-token and joint-localization branches. A differentiable geometric solve recovers in-plane translation while preserving scene, object, and motion priors.

Hand-overlay rendering

  • ViDiHand finetunes only the VACE branch of pretrained Wan2.1-VACE [11] [26] with hand-overlay rendering while freezing the base DiT, preserving scene, object, and interaction priors.The rendering objective includes fully occluded frames and uses no MANO-parameter supervision during adaptation.Source: §3.1 · p. 3, §3.2 · p. 4

Feature extraction

  • The decoder uses a single mid-block activation at L⋆=15 and denoising step τk≈0.7, selected to retain both joint-localization resolution and articulated-pose abstraction.Features are extracted from the 1.3B-parameter Wan2.1-VACE backbone and decoded without backpropagating through the diffusion model.Source: §3.2 · p. 5

Dual-branch MANO decoder

  • Two complementary decoder branches recover the two-hand state: slot-aware hand tokens model holistic articulated MANO pose, while joint heatmaps anchor local image-space coordinates.Mutual cross-attention exchanges joint-level localization evidence and articulated-pose context between the branches.Source: §3.3 · p. 5

Spatio-temporal tokenization

  • Ray-space positional embeddings let one decoder handle heterogeneous camera intrinsics without per-dataset specialization.The tokens combine projected diffusion features with learned spatial and temporal positional codes and camera-ray embeddings.Source: §3.3 · p. 5

Mixed-projection head and training objective

  • The mixed-projection head regresses depth and solves in-plane translation in closed form from refined anchors, jointly optimizing MANO, heatmap, offset, and visibility predictions.This differentiable solve avoids root-translation/root-pose ambiguity and is trained with coupled MANO, camera, image, visibility, and temporal losses.Source: §3.3 · pp. 5–6, §3.4 · p. 6

4 Experiments

Across three egocentric benchmarks, ViDiHand substantially outperforms prior methods under coverage-aware evaluation, including held-out HOI4D. Ablations and qualitative comparisons attribute these gains to pretrained video representations, specialized adaptation, and decoder components that preserve occluded-hand states.

Evaluation protocol

  • Coverage-aware metrics charge false negatives with deterministic, method-independent placeholder errors, preventing conservative methods from avoiding difficult occlusion frames.The protocol reports detection, pose, orientation, position, and temporal metrics, with higher-is-better detection scores and lower-is-better remaining scores.Source: §4.1 · pp. 6–7

Quantitative results

  • ViDiHand ranks first on 26 of 27 metrics across ARCTIC, HOT3D, and HOI4D, including ARCTIC FAcc of 0.997 versus WiLoR’s 0.919.The comparison covers detection, pose, orientation, position, and temporal metrics under the reported benchmark protocol.Source: Table 1 · p. 7, §4.2 · p. 7
  • 21.5 mm is ViDiHand’s HOT3D MPJPE-p, improving from the best baseline’s 31.0 mm; ARCTIC EPE-p falls from WildHands’ 50.5 px to 12.4 px.ViDiHand achieves the best MPJPE-p and PA-MPJPE-p on all three benchmarks.Source: §4.2 · pp. 7–8
  • 3.18 mm/frame^2 is ViDiHand’s ARCTIC jitter, four times below Dyn-HaMR’s 12.8, while ViDiHand ranks first on eight of nine held-out HOI4D metrics.ViDiHand achieves this smoothness without a motion prior or test-time optimization, unlike Dyn-HaMR’s SLAM-guided optimization.Source: Table 1 · p. 7, §4.2 · p. 8

Qualitative comparison

  • On severe occlusions and truncations, baselines often drop, swap, or hallucinate hands, whereas ViDiHand recovers both with plausible articulation.The qualitative comparisons cover ARCTIC, HOT3D, and in-the-wild cases involving objects, frame boundaries, shadows, and hand occlusion.Source: §4.4 · p. 9, Figure 3 · p. 9, Figure 4 · p. 9

5 Conclusion

ViDiHand combines a pretrained video diffusion prior, hand-overlay specialization, and dual-branch decoding to recover 4D two-hand motion from egocentric video. It ranks first on all coverage-aware metrics on ARCTIC and HOT3D and eight of nine on held-out HOI4D, while remaining an offline tool limited to 5.5 fps on four A100 GPUs.

Conclusion

  • ViDiHand recovers 4D two-hand motion using a pretrained video diffusion prior, hand-overlay specialization, dual-branch decoding, and closed-form camera translation.The hand-overlay objective specializes the backbone to the MANO surface while preserving scene, object, and motion priors.Source: §5 · p. 9
  • ViDiHand ranks first on every coverage-aware metric on ARCTIC and HOT3D and on eight of nine metrics on held-out HOI4D.It combines near-complete detection coverage with competitive per-hand accuracy and avoids inference-time smoothing.Source: §5 · p. 9

Limitations and future work

  • At 5.5 fps on 4 A100 GPUs, ViDiHand is currently an offline annotation tool, motivating distillation and few-step or autoregressive generators for real-time inference.Training still relies on pose-labeled video, although joint-only annotations can drive the Stage-1a overlay pretext.Source: §5 · p. 10

Supplementary Material

The supplement first formalizes the coverage-aware evaluation protocol and metric definitions, then extends the main-paper analysis with pairwise, per-side, and qualitative comparisons.

§A

  • The supplement begins with a derivation of the coverage-aware evaluation protocol and closed-form definitions for every evaluation metric.This material appears in §A before the dependent results.Source: Supplementary Material · p. 13

§§C and B

  • It extends the main-paper comparison through pairwise pose evaluation on co-predicted hands across all three benchmarks.This extension is described under §§C and B.Source: Supplementary Material · p. 13

§§C, B, and D

  • It also provides a per-side breakdown of detection F1 and thirteen additional qualitative comparison figures.These analyses further expand the benchmark comparisons beyond the main paper.Source: Supplementary Material · p. 13

A Evaluation Protocol and Metric Definitions

The evaluation reports nine metrics across four categories, with a coverage-aware protocol that integrates detection coverage and pose accuracy into a single pose score. Metrics are defined alongside prediction–ground-truth alignment, and all pose metrics use the protocol except Jitter.

Metric Definitions

Evaluation Protocol

  • The evaluation defines each metric, describes prediction–ground-truth alignment, and then introduces the coverage-aware evaluation protocol.The protocol folds false negatives into every pose metric so detection coverage and pose accuracy are captured by a single number.Source: A Evaluation Protocol and Metric… · p. 14

Coverage-Aware Protocol

  • All pose metrics use the coverage-aware protocol, while Jitter opts out because its second-order finite difference requires three contiguous tracked observations.Metrics computed under the protocol carry a “-p” subscript indicating that every present ground-truth hand contributes.Source: Table 6 · p. 14, A Evaluation Protocol and Metric… · p. 14

A.1 Metric Definitions · A.2 Prediction–Ground-Truth Alignment

The paper evaluates detection, coverage-aware 3D pose, absolute spatial, reprojection, and temporal motion quality using nine metrics, all grounded in a consistent 21-joint convention. Predictions and ground truth are aligned per frame through handedness-constrained IoU matching, visibility thresholds, and off-screen exclusion.

A.1 Metric Definitions

  • All evaluation metrics use the same 21-joint MANO ordering, with joint 0 as the wrist and corresponding ground-truth joints denoted J⋆j.This convention is shared by every loss term and evaluation metric.Source: §A.1 · p. 14
  • Detection quality is measured by frame accuracy, recall, and F1, capturing strict frame-level correctness, matched hand recovery, and the balance between false positives and false negatives.Frame accuracy fails for any missed hand, extra prediction, or handedness swap, while recall and F1 summarize instance recovery and error tradeoffs.Source: §A.1 · pp. 14–15
  • Coverage-aware MPJPE-p and PA-MPJPE-p evaluate root-relative 3D articulation, with Procrustes alignment additionally removing global orientation, translation, and scale.Missed hands receive canonical placeholder costs; PA-MPJPE-p deliberately uses raw placeholder error so false negatives remain costly.Source: §A.1 · pp. 15–16
  • GO-p, CT-p, and EPE-p measure absolute wrist orientation, camera-frame position, and 2D reprojection accuracy rather than root-relative pose.These metrics use geodesic rotation error, Euclidean translation error, and masked pinhole-projected joint distances, respectively.Source: §A.1 · p. 16
  • Jitter measures second-order finite-difference acceleration in mm/frame2 over contiguous true-positive runs of at least three frames, so lower values indicate smoother predictions.It is computed only on true positives because false negatives break identity-continuous runs and cannot support temporal finite differences.Source: §A.1 · p. 17

A.2 Prediction–Ground-Truth Alignment

  • Per-frame alignment handles zero, one, or two hands by greedily matching projected predicted and ground-truth boxes by descending IoU after 10% ground-truth-box dilation.Matches require equal handedness and IoU > 0.1, a deliberately permissive threshold for ambiguous near-misses.Source: §A.2 · p. 17
  • A matched prediction is a true positive when its hand-presence probability exceeds 0.5, with protocol-specific handling for confidence-based and unconditional tracker outputs.The method thresholds σ(ê) > 0.5, baselines use reported probabilities, and tracker emissions are treated as positive.Source: §A.2 · p. 17
  • Ground-truth hands are excluded as off-screen when none of their 21 joints projects inside the image with z > 0.01 m, preventing invisible hands from penalizing evaluation.Using joints rather than mesh vertices aligns the criterion with MPJPE and EPE and avoids counting a fingertip sliver as full visibility.Source: §A.1 · p. 15, §A.2 · p. 18

A.3 Coverage-Aware Evaluation Protocol

The protocol evaluates every ground-truth hand instance, including missed detections, by assigning false negatives canonical MANO placeholders. This yields metrics that jointly reflect detection coverage and pose accuracy rather than rewarding conservative detectors.

Motivation

  • Coverage-aware evaluation includes every ground-truth hand instance, preventing methods with high per-prediction accuracy but lower recall from appearing artificially superior.Standard pose metrics computed only on true positives can silently discard false negatives and reward conservative detectors.Source: §A.3 · p. 18

Placeholder construction

  • For each false negative, the protocol substitutes a canonical MANO output with identity rotation, zero pose, mean shape, and camera-origin placement.Source: §A.3 · p. 18
  • The canonical placeholder yields about 132 mm MPJPE on ARCTIC after wrist anchoring, with absolute depth canceled by the root-relative metric.Source: §A.3 · p. 18

Metric definition

  • Placeholder error is computed per missed sample because it depends on the ground-truth hand’s pose, orientation, and position, with metric-specific definitions provided in §A.1.Source: §A.3 · p. 18

Metric behavior

  • The resulting single metric jointly penalizes missed hands through a substantial deterministic placeholder cost and penalizes inaccurate predictions, making recall failures visible.Source: §A.3 · p. 18

B Per-Side Hand Detection Analysis

ViDiHand maintains per-side hand detection F1 at or above 0.981 on every benchmark, while baseline asymmetry is most severe on HOI4D because single-handed clips trigger spurious predictions for the absent side.

Per-Side Hand Detection

  • On HOI4D, five baselines reach only 0.385–0.473 left-hand F1, reflecting severe asymmetric detection errors.Most HOI4D clips depict single-handed manipulation, causing these pipelines to emit spurious predictions on the absent side.Source: B Per-Side Hand Detection Analysis · p. 18
  • At or above 0.981 per-side hand detection F1, ViDiHand performs consistently across all three benchmarks.Table 7 reports left-hand, right-hand, and combined detection F1, with higher values better.Source: B Per-Side Hand Detection Analysis · p. 18, Table 7 · p. 19
  • Table 7 evaluates left-hand, right-hand, and combined detection F1 across the three benchmarks to isolate left–right confusion.The analysis treats asymmetric per-side F1 as the measure of left–right confusion rather than using a separate handedness metric.Source: B Per-Side Hand Detection Analysis · p. 18, Table 7 · p. 19

C Pairwise Comparison on Co-Predicted Hands · D Additional Qualitative Comparisons

The co-predicted-hands comparison favors baselines by excluding difficult missed-hand cases, with their strongest exceptions occurring under Procrustes-aligned MPJPE and broader wins on held-out HOI4D. Qualitative comparisons across benchmark and in-the-wild clips show ViDiHand recovering occluded, cropped, and difficultly viewed hands with more stable position and articulation than the baselines.

C Pairwise Comparison on Co-Predicted Hands

  • The co-predicted-hands protocol is structurally favorable to baselines because it excludes difficult hands that they fail to detect, including severe occlusion, truncation, blur, and peripheral views.Tables 8–10 compare only frame–hand-side pairs where both methods predict the same-side hand and the ground-truth hand is visible.Source: C Pairwise Comparison on Co-Predicted… · p. 20, Table 8 · p. 20, Table 9 · p. 20
  • On in-distribution benchmarks, baseline wins concentrate on Procrustes-aligned MPJPE, where local articulation is evaluated after removing global rotation, translation, and scale.The strongest crop-based and SLAM-refined baselines can match or surpass ViDiHand under this alignment on hands they successfully detect.Source: C Pairwise Comparison on Co-Predicted… · pp. 20–21
  • On held-out HOI4D, baseline wins broaden across Procrustes-aligned MPJPE, 2D reprojection, and camera translation, consistent with a wider cross-dataset gap.The held-out ViDiHand model receives no explicit HOI4D supervision during training.Source: C Pairwise Comparison on Co-Predicted… · p. 21
  • The held-out designation excludes explicit HOI4D supervision but does not establish that the internet-scale pretrained video backbone never encountered HOI4D.An in-domain HOI4D readout is presented only as a capacity reference, not as a held-out comparison.Source: C Pairwise Comparison on Co-Predicted… · p. 21, Table 11 · p. 21

D Additional Qualitative Comparisons

  • Qualitative comparisons span ARCTIC, HOT3D, and in-the-wild clips using projected joints, MANO overlays, and three novel-view 3D renderings, omitting ground-truth MANO for in-the-wild examples.The figures compare all eight baselines with ViDiHand frame by frame.Source: D Additional Qualitative Comparisons · p. 21
  • Across ARCTIC examples with hands hidden behind objects or cropped by the frame, ViDiHand alone repeatedly recovers the full bimanual configuration with plausible relative positioning and articulation.Baselines commonly miss one or both hands, while Dyn-HaMR may predict both but distort articulation, depth, or relative placement.Source: Figure 5 · p. 22, Figure 6 · p. 22, Figure 7 · p. 23
  • Across HOT3D and in-the-wild cases involving occlusion, fisheye periphery, dim or uneven lighting, blur, and truncation, ViDiHand more consistently preserves hand detection, position, depth, orientation, and articulation.The comparisons include cup, pitcher, bowl, shelf, canister, and single-hand interactions where baselines miss hands, duplicate detections, or produce blobs and depth errors.Source: Figure 9 · p. 24, Figure 10 · p. 24, Figure 11 · p. 25
  • In-the-wild comparisons further show ViDiHand recovering plausible bimanual grasps and avoiding lighting-induced duplicate detections that affect several baselines.Other methods variously miss occluded hands, collapse them into blobs, misplace them in depth, or bend fingers incorrectly.Source: Figure 15 · p. 27, Figure 16 · p. 27, Figure 17 · p. 28

E Implementation Details

The implementation specifies the full pipeline’s data, decoder, losses, and two-stage training procedure. Its video backbone is Wan2.1-VACE [26] [11], a 1.3-billion-parameter diffusion transformer conditioned on egocentric video.

Backbone

  • Wan2.1-VACE [26] [11] provides the video backbone as a 1.3-billion-parameter video diffusion transformer with a VACE conditioning branch.The VACE branch injects the egocentric input video as a conditioning signal.Source: E Implementation Details · p. 29
  • The backbone contains 30 transformer blocks, with L15 denoting the 15th block used throughout the implementation.Source: E Implementation Details · p. 29

Denoising notation

  • The normalized denoising step τk spans τ=0 for pure noise to τ=1 for fully denoised states.This notation corresponds to τk in the main paper.Source: E Implementation Details · p. 29

E.1 Datasets and Cameras … E.5 Controlled Fitting Study: Capacity of the Feature Slice

The appendix specifies ViDiHand’s datasets, camera-aware decoder, losses, and two-stage training pipeline, then verifies that the selected mid-denoise feature slice can support hand-motion decoding. It also documents evaluation scope, including held-out HOI4D testing and baseline training exposure on ARCTIC and HOT3D.

E.1 Datasets and Cameras

  • ViDiHand uses 81-frame segments encoded into 21 latent frames, with dataset-specific spatial patch grids after resizing; HOI4D is held out for testing while ARCTIC and HOT3D provide supervised finetuning data.The datasets differ in camera field of view, frame rate, occlusion patterns, and hand-count conditions; baseline exposure on ARCTIC and HOT3D is explicitly tracked for fair comparison.Source: §E.1 · pp. 29–30, Table 14 · p. 30

E.2 Decoder Architecture

  • The 37-million-parameter decoder combines hand-token cross-attention, joint heatmaps, mutual hand–joint fusion, visibility prediction, and MANO regression in one forward pass.The regression branch predicts orientation, finger rotations, shape, and log-depth, while the heatmap branch supplies per-joint image evidence.Source: §E.2 · pp. 30–31

E.2.1 Ray-Space Positional Encoding

  • Ray-space positional encoding augments learned spatial features with camera-ray information derived from intrinsics, while zero initialization preserves the original spatial encoding at startup.The encoding is computed per token and sample, enabling camera-aware corrections while beginning from a spatial-only representation.Source: §E.2.1 · p. 32

E.2.2 Mixed-Projection Camera-Translation Head

  • The Mixed-Projection Head regresses depth but solves in-plane translation analytically from refined heatmap anchors using valid on-screen joints, producing a differentiable, adaptive least-squares estimate.The decoupled solve uses all eligible joints and propagates translation-loss gradients through both MANO predictions and the heatmap branch.Source: §E.2.2 · p. 32

E.3 Loss Functions

  • The decoder loss combines manifold-aware rotation supervision, translation, shape, visibility, 3D and 2D geometric terms, heatmap supervision, acceleration smoothness, and temporal shape consistency.Geodesic rotation losses respect SO(3), direct heatmap supervision anchors the translation solve, and second-order smoothness penalizes jitter without suppressing smooth motion.Source: Table 15 · p. 33, §E.3 · pp. 33–35

E.4 Two-Stage Training Pipeline

  • Training adapts the video model in two stages: joint-overlay pretraining on EgoDex, followed by mesh-overlay finetuning on ARCTIC and HOT3D, while HOI4D remains excluded from all supervised stages.Stage 1a uses EgoDex joint annotations without MANO labels; Stage 1b continues from that checkpoint with segment-proportional ARCTIC/HOT3D sampling.Source: §E.4 · p. 35, §E.4.1 · p. 35, §E.4.2 · p. 35

E.4.3 Stage 2: MANO Decoder Training

  • The MANO decoder is trained separately on a fixed Stage 1b feature slice: DiT L15 activations at denoising step 34, approximately τk≈0.7, extracted from one 50-step sampling pass.The resulting tensor has 1536 channels, 21 latent frames, and dataset-dependent spatial dimensions; batches are drawn from one dataset at a time to align grids.Source: §E.4.3 · pp. 35–36

E.5 Controlled Fitting Study: Capacity of the Feature Slice

  • A controlled single-segment ARCTIC fitting study tests whether the frozen default mid-denoise slice contains sufficient hand information for accurate and temporally consistent MANO decoding.The dual-branch decoder is fit with the full loss at L15 and τk≈0.7; after 3 k steps it reaches an MPJPE-p value beginning with 0 in the supplied passage.Source: §E.5 · p. 36

F Loss-Term Ablation on ARCTIC

On ARCTIC, removing any single decoder loss worsens pose accuracy, while specific terms control temporal smoothness and rotation consistency. The ablation uses a fixed Stage-1b VACE backbone and restricts only Stage-2 decoder training to ARCTIC.

Experimental setup

  • The ablation isolates decoder effects by training Stage 2 only on ARCTIC while keeping the Stage-1b VACE backbone fixed from the ARCTIC + HOT3D setup.This follows the main-paper ablation protocol and separates decoder ablation from cross-dataset transfer.Source: F Loss-Term Ablation on ARCTIC · p. 36

Loss-term ablation

  • Removing any single loss term increases MPJPE-p versus the full objective; shape consistency causes the smallest increase (0.12 mm), while geodesic rotation causes the largest (0.84 mm).Each ablation removes one term while holding the remaining objective and training protocol fixed.Source: Table 16 · p. 36, F Loss-Term Ablation on ARCTIC · p. 36
  • Removing acceleration smoothness raises Jitter from 3.42 to 3.88 mm/frame2, identifying it as the only loss that materially changes this metric.The result confirms acceleration smoothness acts as a temporal regularizer.Source: F Loss-Term Ablation on ARCTIC · p. 36

Related papers

Loading 2606.30308v2…