PaperlayerView exact revision on arXiv

Source-linked AI summary

DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model

Xueyi Liu, He Wang, Li Yi

arXiv:2510.08556v1cs.ROcs.CV

TL;DR

Generalized in-hand rotation is limited by the sim-to-real gap in contact-rich dexterous dynamics and by constrained prior demonstrations. DexNDM combines a joint-wise neural dynamics model, autonomous data collection, and residual policy adaptation to transfer a simulation-trained policy. The resulting policy generalizes across challenging objects, wrist orientations, and rotation axes, while supporting complex teleoperation tasks.

  • Problem

    General-purpose real-world in-hand rotation remains difficult because contact-rich dynamics create a sim-to-real gap and prior methods cover constrained objects, sizes, wrist poses, or hardware.

  • Method

    DexNDM factorizes dynamics across joints using proprioceptive histories, combines this model with autonomous real-world data collection and residual adaptation, and distills specialist policies into a generalist.

  • Results

    The single policy generalizes to complex, small, and high-aspect-ratio objects across diverse wrist orientations and rotation axes, with simulation gains of 37%–81% over strong baselines.

  • Takeaways & Limitations

    The framework enables broad real-world dexterous rotation and a teleoperation system for tool use and assembly.

  • Takeaways & Limitations

    The model’s performance ceiling is restricted by partial observations, motivating richer hand–object signals and tactile integration.

Abstract

from arXiv · show

Achieving generalized in-hand object rotation remains a significant challenge in robotics, largely due to the difficulty of transferring policies from simulation to the real world. The complex, contact-rich dynamics of dexterous manipulation create a "reality gap" that has limited prior work to constrained scenarios involving simple geometries, limited object sizes and aspect ratios, constrained wrist poses, or customized hands. We address this sim-to-real challenge with a novel framework that enables a single policy, trained in simulation, to generalize to a wide variety of objects and conditions in the real world. The core of our method is a joint-wise dynamics model that learns to bridge the reality gap by effectively fitting limited amount of real-world collected data and then adapting the sim policy's actions accordingly. The model is highly data-efficient and generalizable across different whole-hand interaction distributions by factorizing dynamics across joints, compressing system-wide influences into low-dimensional variables, and learning each joint's evolution from its own dynamic profile, implicitly capturing these net effects. We pair this with a fully autonomous data collection strategy that gathers diverse, real-world interaction data with minimal human intervention. Our complete pipeline demonstrates unprecedented generality: a single policy successfully rotates challenging objects with complex shapes (e.g., animals), high aspect ratios (up to 5.33), and small sizes, all while handling diverse wrist orientations and rotation axes. Comprehensive real-world evaluations and a teleoperation application for complex tasks validate the effectiveness and robustness of our approach. Website: https://meowuu7.github.io/DexNDM/

1 INTRODUCTION

DexNDM targets general-purpose in-hand rotation across diverse objects, wrist orientations, and rotation axes by addressing the sim-to-real gap with joint-wise dynamics and autonomous data collection. The resulting single policy generalizes to challenging real-world objects and supports complex dexterous teleoperation tasks.

  • DexNDM addresses the lack of robust in-the-air rotation across complex shapes, high aspect ratios, varied sizes, wrist orientations, and rotation axes.Prior methods remain constrained by object diversity, hand orientation, sensing hardware, or only one axis of generality.
  • The framework learns joint-wise neural dynamics and uses autonomous real-world data collection to adapt a simulation-trained policy.Joint-wise factorization uses each joint’s proprioceptive history while compressing system-wide influences into low-dimensional net effects.
  • 37%–81%: the base policy outperforms strong baselines on unseen, complex shapes in simulation.The method is validated in simulation and the real world, with simulation results showing broad object generalization.
  • Real-world sim-to-real adaptation significantly and consistently improves rotation across diverse wrist orientations, rotation axes, and broad object distributions.The evaluated distribution includes complex geometries, aspect ratios up to 5.33, and object-to-hand ratios of 0.31–1.68.
  • The policy rotates challenging objects with a smaller LEAP hand, including animal-shaped objects and long objects in a downward-facing configuration.The paper reports rotation of 10–16 cm objects around their long axis for about one full circle in the air, and performance matching or surpassing Visual Dexterity on several shapes.
  • The general rotation policy enables teleoperation for complex dexterous tasks such as tool use and assembly.The paper also presents an ablation study of the dynamics-model and data-collection design choices.

3 METHODOLOGY

DexNDM builds a deployable generalist rotation policy by distilling category specialists, then adapting it to real-world dynamics with joint-wise modeling, autonomous data collection, and a residual policy. The approach uses per-joint histories and low-dimensional effective terms to improve generalization while reducing data and modeling burdens.

  • Specialist-to-Generalist and Sim-to-Real: The pipeline distills category-specific RL specialists into one generalist policy, then augments it with a residual for sim-to-real adaptation.The generalist is trained with behavior cloning from privileged-observation oracle policies.
  • Joint-Wise Neural Dynamics: Joint-wise dynamics predicts each joint’s next state from its own W-step state-action history while compressing system-wide effects into low-dimensional effective terms.These terms represent influences such as inter-joint coupling, actuation, and object-induced effects.
  • Joint-Wise Neural Dynamics: The joint-wise representation acts as an information bottleneck that discards spurious correlations while retaining enough information to predict joint evolution.Its low dimensionality also prevents reconstructing irrelevant high-dimensional system-wide influences.
  • Theoretical Rationale: Projection onto joint histories contracts distribution shift and yields tighter generalization than whole-hand modeling under the stated assumptions.The paper formalizes this through KL-divergence contraction and a smaller target-domain generalization gap.
  • Autonomous Data Collection: The Chaos Box collects diverse loaded interactions autonomously by replaying simulated actions in soft balls, optionally adding Gaussian action noise, without human resets.The strategy uses policy awareness, object-loaded interaction, broad coverage, and scalability.

4 EXPERIMENTS

The experiments evaluate generalization in simulation, real-world rotation across challenging objects and wrist conditions, comparisons with prior methods, and the dynamics model and data-collection design choices.

  • Real World Results: Real-world evaluations test long, small, and complex objects under challenging wrist poses, including multi-axis and multi-wrist-orientation rotation.The experiments use a LEAP hand and report rotation and time-to-fall metrics across real-world settings.
  • Simulation Results: The policy generalizes to unseen objects across randomized wrist orientations and four rotation-axis groups, outperforming a re-implemented baseline in simulation.Evaluation uses ContactDB objects and axis groups ±x, ±y, ±z, and 26 general axes.
  • Comparisons to AnyRotate: The method substantially outperforms AnyRotate on four replicable objects and supports smaller objects and higher aspect ratios than the compared setup.The comparison uses rotation degrees and time-to-fall under two AnyRotate test settings.
  • Comparisons to Visual Dexterity: Compared with Visual Dexterity, the evaluation uses survival rotation angles because the task definitions differ, and reports comparable or superior results despite occasional table support for Visual Dexterity.The metric estimates how many 90-degree intervals an object rotates before falling.
  • Comparisons to Whole-Hand Neural Dynamics: The joint-wise model is nearly as expressive in high-data in-domain settings, more sample-efficient with limited data, and more transferable across training distributions than the whole-hand model.Figure 6 compares 3.1M and 7.5k data regimes and transfer from different training distributions.
  • Ablations and Data Collection: Ablations examine dynamics-model components and autonomous data-collection strategies through generalization error, real-world performance, collection efficiency, dataset size, and scaling behavior.These studies are conducted with the palm fixed, z-axis rotation, and a consistent wrist pose.

6 CONCLUSIONS AND LIMITATIONS

The paper concludes that joint-wise neural dynamics and autonomous data collection enable dexterous rotation of challenging objects. It identifies partial observations and the resulting ceiling on model performance as a limitation, with richer hand–object signals and tactile sensing as future directions.

  • Conclusion: The framework enables dexterous rotation of challenging objects through a joint-wise neural dynamics model and autonomous data collection.The conclusion presents these components as the center of the sim-to-real framework.
  • Limitations: Partial observations restrict the model’s performance ceiling, motivating richer hand–object transition signals and tactile integration.The paper identifies these as valuable future directions.

A ADDITIONAL EXPLANATIONS OF THE METHOD

The method section specifies the policy’s observations, rewards, control interface, architecture, and specialist-to-generalist training design. It combines proprioceptive and contact information with residual policy learning and category-specific teacher policies distilled into one deployable policy.

  • Observations: The policy observes histories of joint positions and targets, velocities, fingertip and object states, forces, contacts, wrist orientation, and the target rotation axis.The oracle observation includes the listed proprioceptive, object, force, contact, wrist, and axis signals.
  • Rewards: The reward combines rotation, goal, and penalty terms, encouraging target-axis rotation while penalizing off-axis motion, pose deviation, object translation, work, and torque.The rotation term clips angular velocity along the unit target axis, and the goal term supplies an intermediate orientation target.
  • Control Strategy: Torque control runs at 20 Hz through repeated low-level executions, with joint torques computed from target-position error and velocity damping.The controller uses fixed proportional and derivative gains.
  • Policy Architecture: The generalist policy is a residual MLP with five 1024-dimensional residual blocks mapping observations to outputs.Each block combines three neural-network transformations with ReLU activations.
  • Training Design: The training design combines category-specific oracle policies into a unified behavior-cloning policy for real-world deployment and generalization across diverse objects.The paper leaves oracle training as a collection of specialist policies rather than training one fully general teacher.

A.2 PROOF OF MAIN THEOREMS

The proofs establish that lower-dimensional representations reduce distributional divergence and can improve target risk when approximation error is outweighed by the reduced generalization gap. These conditions support joint-wise modeling under data scarcity and train–test shift.

  • Distributional bounds: The data processing inequality gives KL(P∥Q) ≥ KL(g(P)∥g(Q)), with strictness when the projection merges points whose conditional distributions differ.The proof characterizes strictness through a positive conditional KL divergence after the representation transformation.
  • Distributional bounds: Because functions of g_X discard information, the induced function class is smaller, yielding a strict IPM inequality when the optimal full-space discriminator cannot depend only on g_X.The proof uses F2 ⊆ F1 and shows the maximizing function for F1 is unavailable in F2.
  • Target-risk proposition: If reduced-space approximation error is ϵA and its generalization-gap advantage is ϵB with ϵB > ϵA, its target risk is strictly lower than the full-space predictor’s.The result is expressed as RP(f_Q2 ◦ g_X) < RP(f_Q1).
  • Validity in dexterous manipulation: Experiments support the assumptions: joint-wise modeling preserves in-domain performance with sufficient data, improves sample efficiency in low-data regimes, and transfers better under distribution shift.The authors report these patterns across the experiments in Section 4.2 and Figure 6.
  • Validity in dexterous manipulation: Data scarcity and train–test shift are pervasive in dexterous manipulation, while whole-hand dynamics degrade sim-to-real transfer in the reported tables.The paper contrasts this setting with prior whole-body modeling used in an in-distribution, less complex dynamics regime.

A.3 RATIONALITY OF JOINT-WISE DYNAMICS MODELING (PART I)

Joint-wise dynamics models each joint from its own state–action history while treating coupled effects as low-dimensional effective terms. Smoothness analyses support this approximation over short windows for joint states, active torques, and external-torque proxies.

  • Model formulation: The manipulator dynamics rewrite isolates an effective external torque for the modeled joint, absorbing coupled-joint and object effects into a scalar effective term.The derivation introduces τ_eff and separates it from the modeled joint’s applied torque.
  • Model formulation: Joint-wise dynamics predicts one joint’s transition from its own history, implicitly accounting for time-varying influences over a short 10-frame, 0.5-second window.The model relies on short-window continuity of slave-joint states, active torques, and effective external torque.
  • State and action smoothness: Joint-state trajectories are well approximated by low-order polynomials, supporting the continuous-function assumption used for short-window prediction.Fifth-order fits are reported as excellent after satisfactory third-order fits.
  • State and action smoothness: Active-torque trajectories remain satisfactorily fitted despite being more complex than joint states, with fit quality improving as polynomial order increases.The active torque is estimated from the difference between positional targets and joint states.
  • State and action smoothness: Virtual external-torque trajectories also admit satisfactory polynomial fits, although their evolution is more complex than state and active-torque trajectories.The virtual force is estimated from the difference between actual actions and inverse-dynamics actions learned from free-hand trajectories.

A.4 RATIONALITY OF JOINT-WISE DYNAMICS MODELING (PART II)

Experiments show that a joint’s history predicts its own transition but cannot recover neighboring joints’ states or actions, supporting implicit low-dimensional net-effect modeling rather than reconstruction of full coupling. This factorization improves transfer, with a stated in-domain and training-speed trade-off.

  • Predictability of coupling: The failure to predict neighboring states and actions indicates that single-joint history does not contain enough information to reconstruct complex high-dimensional coupling effects.The authors therefore interpret the model as learning net effects implicitly rather than underlying coupling factors individually.
  • Predictability of coupling: Predicting neighboring joints from one joint’s history has generalization error three orders larger than predicting that joint’s own transition.Even in-distribution validation errors for neighboring-joint predictions remain two orders larger than own-transition prediction.
  • Interpretation: Per-joint histories capture comparatively stable low-dimensional net effects, enabling transfer across changes in whole-hand interaction while avoiding system-wide variation.The conclusion links this stability to the observed expressiveness, sample efficiency, and generalization of joint-wise dynamics.
  • Limitations: The joint-wise model performs slightly worse than the whole-hand model in-domain under the multi-task high-data regime and requires longer training because joints are processed iteratively.These are the stated limitations of the modeling choice.

A.5 COMPARISONS OF DATA DISTRIBUTIONS BETWEEN COLLECTED TRAJECTORIES AND ROTATION TRAJECTORIES

The distribution analysis compares autonomous collection with task-relevant rotation trajectories and finds that per-joint data cover the task distribution, whereas per-finger and whole-hand distributions diverge substantially.

  • Distribution comparison: The comparison uses 20 cube-rotation trajectories collected under the Thumb Up wrist orientation, totaling approximately 8,000 task-relevant data points.Figures 15–17 summarize per-joint, per-finger, and whole-hand distributions against these task-aware trajectories.
  • Distribution comparison: Per-joint state–action trajectories from autonomous collection cover the distribution of task-aware rotation trajectories.This coverage is reported for the per-joint representation rather than the aggregated representations.
  • Distribution comparison: Per-finger and whole-hand distributions show a large discrepancy from the task-relevant rotation distribution.The result contrasts with the coverage observed at the per-joint level.

B.1 TRAINING PERFORMANCE

The method achieves stronger training performance than re-implemented AnyRotate across different training sets, while prior Hora-based approaches struggle even on basic cylinder objects and down-facing hands.

  • B.1 TRAINING PERFORMANCE: Prior Hora-based policies struggle to achieve satisfactory results even on basic cylinder objects and fail in the down-facing hand scenario.
  • B.1 TRAINING PERFORMANCE: The comparison therefore tests training behavior across both standard and irregular object distributions rather than a single object category.
  • B.1 TRAINING PERFORMANCE: Training performance is compared against re-implemented AnyRotate across multiple training sets, including irregular “DexEnv Objects.”Figure 18 reports final total reward for both methods across these sets.

B.2 ADDITIONAL REAL WORLD RESULTS

Additional real-world results demonstrate rotation in the air under diverse wrist orientations, with qualitative pose-tracking evidence included for the reported experiments.

  • B.2 ADDITIONAL REAL WORLD RESULTS: Real-world results show the system rotating challenging objects in the air.
  • B.2 ADDITIONAL REAL WORLD RESULTS: The evaluation includes diverse wrist orientations, extending beyond a single fixed hand pose.

B.3 CASE STUDY ON THE EFFECTIVENESS OF OUR SIM-TO-REAL METHOD

DexNDM improves rotation on challenging shapes relative to the base policy alone, enabling longer rotations and greater stability for difficult object geometries and poses.

  • B.3 CASE STUDY ON THE EFFECTIVENESS OF OUR SIM-TO-REAL METHOD: DexNDM produces superior rotation performance over the base policy without sim-to-real adaptation on challenging shapes.Table 7 covers high aspect ratios, small sizes, and complex geometry in a down-facing-hand setting, averaging three trials.
  • B.3 CASE STUDY ON THE EFFECTIVENESS OF OUR SIM-TO-REAL METHOD: The residual policy extends rotation of a 3 cm × 13.5 cm Lego leg from at most 180 degrees to almost a complete circle.
  • B.3 CASE STUDY ON THE EFFECTIVENESS OF OUR SIM-TO-REAL METHOD: For a thin 3 cm × 3 cm × 10 cm cuboid, the residual policy improves stability during long-term rotation despite using an unstable three-finger gait.

B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES

The analysis contrasts residual-policy adaptation with direct fine-tuning, examines autonomous and task-relevant data collection, and documents practical limitations in pose tracking and data quality.

  • B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES: Direct fine-tuning on the learned dynamics model is unstable and hyperparameter-sensitive, failing to execute even basic rotations.
  • B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES: Table 8 reports per-joint delta-action magnitudes during real-world cylinder rotation to quantify compensation applied to each joint.
  • B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES: Task-relevant pose-annotated collection is limited by occlusions, symmetric-object ambiguity, noisy tracking, lengthy setup, and retention of only successful trajectories.The resulting data are biased toward objects and scales compatible with training and pose estimation.
  • B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES: Small objects cause pose-tracking failure during hand transfer, while axis-symmetric objects produce unstable pose estimates that rotate despite remaining still.
  • B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES: Autonomous data collection is object-agnostic, supports diverse load influences and replay-action noise, and requires no human intervention.
  • B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES: Wave-based data collection can cause self-collision, distribution shift, and laborious frequency and magnitude design, motivating policy-rollout data.
  • B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES: Pose-annotated baseline training for ASAP and UAN failed to converge, with no meaningful reward improvement, likely because of poor data quality.

C ADDITIONAL EXPERIMENTAL DETAILS

The experiments detail object sets, randomization, baseline comparisons, failure analyses, and autonomous data-collection constraints underlying DexNDM’s real-world evaluation.

  • Comparisons: The evaluation compares DexNDM with AnyRotate on shared rotation axes and wrist orientations, reporting performance averaged over three independent trials.The comparison uses four reproducible AnyRotate objects and the same rotation-axis and hand-orientation settings.
  • Comparisons: DexNDM rotates smaller objects and higher-aspect-ratio objects than AnyRotate, including objects as small as 3cm and aspect ratios up to 5.3.The reported examples include vegetable models, a Lego leg at 4.5, a book at 5.3, and a long cuboid at 3.33.
  • Failure analysis: Baseline sim-to-real methods failed on an easy cylinder because compensators trained on free-hand dynamics did not generalize to object-induced dynamics.The authors hypothesize an out-of-distribution issue and connect the failure to the importance of modeling object dynamics.
  • Failure analysis: Training ASAP and UAN on 54 task-relevant trajectories failed to converge, which the authors attribute to limited data and noisy object-state annotations.The reward showed little to no improvement during compensator training.
  • Ablation: DexNDM’s residual-policy compensation remained functional with lower-quality free-hand data, whereas baseline fine-tuning strategies could totally fail under inaccurate learned dynamics.The ablated joint-wise model rotated a basic cylinder for at least one circle but did not surpass the base policy.
  • Data collection: Autonomous collection has practical limits: the Chaos Box cannot collect palm-up data, while fixed-ball perturbations are less diverse and alternative setups require manual intervention.Finger-mounted objects could fall, and table-supported objects could leave the robot’s workspace.
Loading 2510.08556v1…