PaperlayerView exact revision on arXiv

Source-linked AI summary

DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction

Wenliang Guo, Zhanbo Huang, Yu Kong

arXiv:2609.36454v1cs.CV

TL;DR

Monocular HOI videos provide partial evidence, and existing methods mainly constrain visual and geometric agreement rather than interaction dynamics. DynamicHOI combines geometry-grounded diffusion refinement with coupled hand-object dynamics and an actuation prior. It improves hand and object reconstruction across three datasets, with reported benefits for downstream applications.

  • Problem

    Occlusion and viewpoint ambiguity in monocular video leave multiple plausible 3D trajectories, while visual and geometric supervision does not explicitly constrain interaction dynamics.

    Source: §1 · pp. 1–2

  • Method

    DynamicHOI grounds visual evidence in hand-object geometry, derives hand forces and object wrenches for dynamics supervision, and couples them through contact with a probabilistic prior on active hand actuation.

    Source: §1 · p. 2

  • Results

    DynamicHOI consistently improves hand and object reconstruction across three HOI datasets, and reconstructed trajectories benefit world-model generation and robotic manipulation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that physics-aware reconstruction provides more useful motion signals beyond reconstruction accuracy, including for world modeling and dexterous manipulation.

    Source: §1 · p. 2, §5 · p. 10

    The framework's object representations are tied to known template meshes; future work aims to generalize to unseen objects.

    Source: §5 · p. 10

Abstract

from arXiv · show

We study hand-object interaction (HOI) reconstruction from monocular RGB videos, where partial observations can produce visually plausible yet mechanically inconsistent trajectories. Existing methods mainly enforce visual and geometric agreement, leaving the underlying interaction dynamics insufficiently constrained. We propose DynamicHOI, a physics-aware HOI reconstruction framework combining geometry-grounded diffusion refinement with coupled hand-object dynamics. Geometry spatially grounds visual evidence for trajectory refinement, while articulated inverse dynamics and Newton-Euler dynamics derive hand generalized forces and object wrenches for dynamics-level supervision. We further couple hand and object dynamics through contact-force transfer and recover active hand actuation as an interaction-level physical quantity. We formulate its empirical magnitude distribution into a probabilistic prior that penalizes unlikely actuation and suppresses mechanically implausible reconstructed motion. Experiments on three HOI datasets show consistent improvements in both hand and object reconstruction. The reconstructed trajectories further benefit downstream applications including hand world-model generation and robotic manipulation learning, demonstrating the value of physics-aware HOI modeling beyond reconstruction.

1 Introduction

Monocular HOI reconstruction is ambiguous, and trajectory-level visual agreement does not ensure mechanically consistent motion. DynamicHOI combines geometry-grounded refinement with coupled hand-object dynamics and an actuation prior; experiments report improvements in reconstruction and downstream applications.

1 Introduction

  • Monocular occlusion and viewpoint ambiguity allow multiple 3D trajectories to explain similar visual evidence, while trajectory-level supervision does not explicitly constrain implied dynamics.A reconstructed object may translate or rotate without a corresponding hand force.Source: §1 · p. 1
  • Prior articulated-hand dynamics derive generalized forces from trajectories, but do not account for manipulated-object dynamics or contact coupling between hand and object.The hand and object are mechanically linked: active hand actuation produces motion and contact forces that drive the object.Source: §1 · pp. 1–2
  • DynamicHOI spatially grounds visual evidence in hand-object geometry during diffusion refinement and aligns motion-implied hand forces and object wrenches for dynamics-level supervision.It derives hand generalized forces through articulated inverse dynamics and object wrenches through Newton-Euler dynamics.Source: §1 · p. 2
  • Contact-force transfer couples hand and object dynamics, while a probabilistic prior penalizes low-probability active hand actuation.The prior is based on an estimated empirical actuation distribution and is intended to suppress mechanically implausible reconstructions.Source: §1 · p. 2
  • DynamicHOI consistently improves hand and object reconstruction across three HOI datasets, with reconstructed trajectories also benefiting world-model generation and robotic manipulation.The authors attribute the downstream gains to more useful motion signals beyond reconstruction accuracy.Source: §1 · p. 2

2 Related Work

Prior HOI methods use visual, geometric, and interaction cues, while physics-aware approaches address selected physical constraints. DynamicHOI grounds visual cues in 3D HOI geometry and jointly models hand-object dynamics through contact.

2 Related Work

  • Existing hand and object reconstruction methods do not explicitly use joint 3D hand-object geometry to organize visual evidence; DynamicHOI grounds cues in that geometry during refinement.Source: §2 · p. 2
  • Joint HOI methods exploit hand-object dependencies, but primarily enforce visual and geometric consistency rather than the physical requirements underlying their trajectories.Source: §2 · p. 2
  • The framework first estimates an initial trajectory in parametric states, progressively refines it with diffusion, and decodes it to an HOI mesh trajectory.Source: Figure 1 · p. 3
  • Prior physics-aware methods address selected constraints, but do not jointly model hand dynamics, object dynamics, and contact coupling; DynamicHOI also regularizes active hand actuation with a probabilistic prior.Source: §2 · pp. 2–3

3 Methodology

DynamicHOI refines hand-object trajectories using geometry-grounded diffusion and constrains their individual and coupled dynamics. Its training objectives combine trajectory alignment, dynamics supervision, and a contact-based actuation prior.

3.1 Problem Formulation

  • The objective combines trajectory alignment with separate hand-object dynamics constraints and a coupled-interaction constraint, balancing the physical terms with weights λd and λa.These constraints are training objectives; trajectory-level supervision alone does not guarantee physically plausible motion.Source: §3.1 · pp. 3–4

3.2.1 Refinement Network

  • A diffusion refinement network progressively denoises initial estimates, using 3D geometry, spatially corresponding visual evidence, and hand-object dependency modeling.Geometric encoding exposes mesh structure; projected hand and object anchors retrieve aligned image features.Source: §3.2.1 · p. 4

3.2.2 Hand and Object Dynamics

  • Hand inverse dynamics derives generalized forces, while Newton–Euler dynamics maps object pose trajectories to wrench profiles for dynamics-level supervision.The object wrench represents the non-gravitational force and torque required for object translation and rotation; mass comes from the dataset when available, otherwise it is estimated using a large language model.Source: §3.2.2 · p. 5

3.2.3 Coupled HOI Dynamics

  • Active hand actuation combines the force required for hand motion with contact forces inferred to explain the object wrench, coupling the two bodies’ dynamics.The method models in-air manipulation with six hand-mesh contact anchors: five fingertips and the palm center.Source: §3.2.3 · p. 6
  • A separate Student-t distribution for each actuation degree of freedom supplies a negative-log-likelihood prior that penalizes unlikely actuation while tolerating occasional high values.The empirical distribution is heavy-tailed; the objective also uses ground-truth contact indicators and emphasizes dynamically active frames.Source: §3.2.3 · pp. 6–7, Figure 2 · p. 6

3.3 Training Objective and Inference

  • End-to-end training combines diffusion trajectory reconstruction, individual hand and object dynamics supervision, and contact-based coupled mechanics; physical quantities are calculated only during training.The method therefore introduces no additional computational cost at inference.Source: §3.3 · p. 7

4 Experiments

Experiments across three HOI datasets assess reconstruction and downstream utility. DYNAMICHOI improves hand and object reconstruction, with benefits also reported for robotic manipulation and egocentric world modeling.

4.1 Experimental Setting

  • Evaluation covers egocentric HOT3D and allocentric HO3D-v2 and DexYCB, using dataset-specific hand metrics and object-pose metrics.The experiments follow dataset-specific splits and protocols.Source: §4.1 · p. 7

4.2 Experimental Results Analysis

  • Across HOT3D, HO3D-v2, and DexYCB, DYNAMICHOI leads on most metrics, with especially large HOT3D gains and lower hand and object ACCEL across all datasets.On DexYCB, it also outperforms PAD-Hand, which explicitly models hand dynamics; the comparison is summarized in Table 1.Source: §4.2 · p. 7, Table 1 · p. 8
  • Module ablations show visual grounding matters most for HOT3D hand reconstruction, while interaction matters most for HO3D hand and HOT3D object reconstruction.On HOT3D, dense hand-object co-motion can reduce the interaction module’s marginal effect for hand reconstruction; on fixed-view HO3D, geometry is less critical.Source: §4.2 · p. 7, Table 2 · p. 8
  • Adding Ldyn substantially improves hand and object reconstruction on HOT3D; Lact adds a smaller but consistent gain, while physical-loss gains are modest on fixed-view HO3D.The authors attribute some limited actuation-prior improvement to video variability and errors in force estimates that may propagate into actuation-distribution inference.Source: §4.2 · pp. 7–8, Table 3 · p. 9
  • On HOT3D, fitting the Student-t degrees of freedom separately for each DoF performs best overall, while better likelihood fit alone does not guarantee better object reconstruction.The Gaussian prior underperforms removing Lact, consistent with the inferred actuation’s heavy-tailed nature.Source: §4.2 · p. 8, Table 4 · p. 10

4.3 Downstream Applications

  • 25.0% to 74.9% success rate (WiLoR to DYNAMICHOI) on dexterous manipulation, while DYNAMICHOI performs best among reconstructed action sources on all world-modeling metrics.Both applications vary the input hand trajectories while keeping other components fixed; the trajectories come from models trained on the corresponding datasets without task-specific retraining.Source: §4.3 · pp. 8–9, Figure 4 · p. 10

5 Conclusion

DYNAMICHOI combines geometry-grounded trajectory refinement with coupled hand-object dynamics and a learned prior over active hand actuation. Experiments show improved reconstruction and downstream gains in dexterous manipulation and egocentric world modeling.

5 Conclusion

  • DYNAMICHOI models articulated-hand and rigid-object dynamics, couples them through contact, and regularizes active hand actuation with a learned probabilistic prior.Source: §5 · p. 10
  • Experiments on egocentric and allocentric datasets demonstrate improved reconstruction, with downstream gains in dexterous manipulation and egocentric world modeling.Source: §5 · p. 10
  • Future work will explore more accurate force estimation, richer actuation priors, and object representations that generalize beyond known template meshes.Source: §5 · p. 10
Loading 2609.36454v1…