PaperlayerView exact revision on arXiv

Source-linked AI summary

WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation

Jie Yin, Zeyuan Zhao, Xiaojing Tan, Yang Liu, Chiyu Wang, Xinyang Gu

arXiv:2609.07002v1cs.RO

TL;DR

Robust in-hand manipulation requires inferring object geometry, pose, contact, and slip from partial, noisy observations, while existing policies can be fragile to pose shifts, disturbances, and object variation. WM-Craftnet learns multimodal action-conditioned recurrent dynamics and uses the resulting state as policy context, with clean-depth reconstruction supervision. The paper reports improved multi-object rotation and transfer, including a reusable prior for 49-object control.

  • Problem

    In-hand policies can be fragile to pose shifts, force disturbances, and object variation, while robust adjustment requires inferring geometry, pose, contact, and slip from partial observations.

    Source: §1 · p. 1

  • Method

    WM-Craftnet trains a multimodal action-conditioned recurrent model with clean-depth reconstruction and reward prediction, then supplies its deterministic state as policy context.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    WM-Craftnet improves multi-object rotation, unseen-object transfer, perturbation recovery, and sim-to-real performance; on the y-axis tool-like set, return rises from 177.8 to 272.7.

    Source: §4.4 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    A WSM pretrained on nine z-axis objects serves as a reusable prior for downstream 49-object policy learning.

    Source: §1 · p. 2, §3.4 · p. 5

    Recovery remains limited under large disturbances.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Generalizable and robust dexterous in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip from partial and noisy observations. Although recent tactile and visuotactile RL methods achieve strong in-hand rotation in controlled settings, their robustness often degrades under pose shifts, force disturbances, and object variation. We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction. Rather than using the world model for latent imagination or policy optimization, WM-Craftnet uses the learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor--critic policy. Importantly, WSM is trained to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment. Ablations over recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads show that predictive world modeling, clean-depth supervision, and tactile contact cues all shape the learned state. A WSM pretrained on nine \(z\)-axis objects serves as a reusable prior for \(49\)-object downstream policy learning. This context improves multi-object rotation, with quantitative and qualitative evidence for unseen-object, perturbation-recovery, and sim-to-real transfer.

1 Introduction

In-hand manipulation must infer object state and contact from partial observations while adapting to pose, geometry, and force changes, but narrow training sets can produce fragile policies. WM-Craftnet uses a predictive multimodal recurrent state for robust rotation and evaluates it across object variation, perturbations, and sim-to-real transfer.

1 Introduction

  • Policies trained on narrow object sets or fixed initial poses can overfit to object-specific finger gaits and become fragile to pose offsets, force disturbances, and object drift.Pose-robust adjustment requires inferring geometry, pose, contact state, and slip risk under partial observability.Source: §1 · p. 1
  • WM-Craftnet learns an action-conditioned recurrent representation from proprioception, noisy depth, tactile sensing, actions, and multimodal prediction targets.Noisy depth inputs are trained against clean wrist-depth reconstruction targets, while reward is also used as a prediction target.Source: §1 · p. 2
  • The paper reports improved multi-object rotation, unseen-object evaluation, perturbation recovery, and sim-to-real transfer, with WSM pretraining reused for 49-object downstream control.The real-world evaluation uses a human-sized 22-DoF five-finger dexterous hand.Source: §1 · p. 2

2 Related Work

Prior dexterous-manipulation work combines vision, touch, proprioception, and learned control in different ways, from axis-wise skill composition to object-state estimation. WM-Craftnet is positioned as learning recurrent interaction dynamics for sustained in-hand control, rather than relying on imagined rollouts for policy optimization.

2 Related Work

  • In-hand approaches include proprioceptive adaptation, vision-based pose estimation [3], binary or dense tactile feedback [5], and composed axis-wise skills [8].Other strategies include real-trajectory adaptation [10], hardware-learned actuator corrections [11], and randomized 3D Gaussians for monocular pose estimation.Source: §2 · p. 2
  • Visuotactile methods use temporal transformers, shared 3D representations [6], neural fields, tactile skins [16], cross-modal features, and contact-aware geometric objectives [18].WM-Craftnet instead learns action-conditioned recurrent dynamics with multimodal prediction objectives and supplies its deterministic state as detached PPO context.Source: §2 · p. 2
  • World-model control includes imagined-trajectory optimization in Dreamer [20] [21], physical-robot applications [22], and neural-simulator policy optimization [24].WM-Craftnet follows predictive-feature conditioning without imagined PPO rollouts, adding noisy-to-clean depth reconstruction and reuse across object distributions [7] [25].Source: §2 · pp. 2–3

3 WM-Craftnet Framework

WM-Craftnet defines a Dreamer-style recurrent latent dynamics model whose detached deterministic state provides deployable policy context for partially observed in-hand rotation. Its multimodal inputs and prediction targets are designed to represent geometry and contact over time, with pretraining and PPO policy learning supporting multi-object control.

3.1 Problem Formulation and Observations

  • The task is continuous in-hand rotation around a commanded axis, using on-robot observations, the axis command, and projected WSM state to produce relative 22-DoF joint-target commands.The deployable observation contains proprioception, tactile/contact measurements, noisy wrist depth, and the previous command, not clean simulator or decoder depth.Source: §3.1 · p. 3

3.2 Multimodal Predictive State Encoder

  • The world model encodes proprioception and tactile/contact with MLP branches and depth with a CNN, then uses recurrent dynamics to preserve temporal information.Noisy augmented depth is reconstructed against clean hand-object depth targets, encouraging denoised geometry from corrupted observations.Source: §3.2 · p. 4

3.3 World Model for In-Hand Manipulation

  • The Dreamer-style RSSM is trained with image, proprioception, tactile, reward, and KL objectives, while its detached deterministic recurrent feature conditions the actor.The world-model targets include clean depth reconstruction and reward prediction; simulator pose and shape labels also supervise auxiliary heads during training.Source: §3.3 · p. 4

3.4 Pretrained Synesthetic Prior for Multi-Object Control

  • WSM pretrained on contact-rich data from nine z-axis objects initializes the 49-object experiment and continues adapting to new rollouts.The resulting recurrent representation captures co-varying depth, proprioception, action, reward, object motion, and contact evidence.Source: §3.4 · p. 5

3.5 World Model-Conditioned Manipulation Policy

  • PPO trains the actor from deployable observations and projected WSM features, while the critic additionally receives simulator state during training.The actor's relative joint-target updates are smoothed and clamped, and stopped gradients decouple the RSSM update from the actor interface.Source: §3.5 · p. 5

4 Experiments and Evaluations

The experiments compare WM-Craftnet with reimplemented baselines under shared simulation and hardware conditions, then test multi-object rotation, WSM mechanisms and reuse, harder rotation axes, and robustness to unseen objects and perturbations. Across these evaluations, the reported results include gains in rotation performance and hardware success, alongside a limited recovery rate under severe disturbances.

4.2 Robust Multi-Object z-Axis Rotation

  • With nine randomized-start objects, pretrained full WSM reaches 753.3 return and 1.293 RotR, while scratch-trained WM-Craftnet reaches 414.3 return and 0.742 RotR.Scratch training also reduces OffAxis and AngVar to 1.285 and 1.735; full WSM reaches 1.225 and 1.324, respectively.Source: §4.2 · p. 6, Table 1 · p. 6, Figure 2 · p. 7

4.3 World Synesthesia Model Analysis

  • Full depth–touch input reaches 753.3 return, versus 684.5 with proprioception alone; the auxiliary-head study reaches 753.3 with all heads versus 688.1 without them.With proprioception and touch, return is 698.9 and OffAxis is 1.165; reducing tactile availability also lowers return from 753.3 to 743.4 at 25% dropout and 724.9 with all-zero input. WSM states show object-related regions and overlap across objects and rollout phases, while depth reconstruction suppresses artifacts and preserves hand-object geometry.Source: §4.2 · p. 6, §4.3 · p. 7, Figure 4 · p. 7
  • Clean-depth supervision raises return from 708.0 to 753.3, while replacing recurrent policy context with a per-step feature lowers it to 667.6.Removing the previous recurrent state gives 705.4 return and 1.282 RotR, but improves OffAxis and AngVar to 1.125 and 1.185.Source: §4.2 · p. 6, Table 2 · p. 6
  • After 3000 epochs of downstream training on 49 new objects, the pretrained WSM policy averages 9.37 ± 0.13 rad per episode versus 3.28 rad without the prior.Training fall rate drops from 6% to 0.3%; in a separate five-block diagnostic, RotR rises from 1.10 from scratch to 1.34 with a 9-object prior and 1.46 with a 49-object prior.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4 More Difficult Rotation Modes

  • On nine tool-like objects, WM-Craftnet raises y-axis return from the strongest baseline’s 177.8 to 272.7 and RotR from 0.545 to 1.025, while achieving the lowest fall rate.The 88.1% RotR gain exceeds the 11.7% episode-length gain, while ObjVel ranks second rather than first.Source: §4.4 · p. 8, Table 4 · p. 8
  • For x-axis rotation, WM-Craftnet reaches 432.8 return and 435.4 episode length, with the highest RotR and lowest fall rate.Blind, tactile-only, and depth-only policies stay below 45 return and 64 episode length; open-loop replay lasts longer but has low return, low RotR, and a high fall rate.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 Stress Tests in Challenging Interaction States

  • Under severe simulated perturbations, recovery success is 14.1% versus 6.2% for the strongest baseline, with a 2.56 s mean recovery time; the absolute recovery rate remains limited.In a separate large-scale hardware protocol, WM-Craftnet succeeds in 175/200 trials across 20 objects.Source: §4.5 · pp. 8–9, Table 6 · p. 9, Figure 6 · p. 9
  • WM-Craftnet achieves the highest RotR and return on held-out objects across all three axes, and succeeds in 175/200 hardware trials across 20 objects versus 33/200–53/200 for baselines.Held-out RotR gains over the strongest non-WSM entries are 8.6%, 89.2%, and 20.9% for x, y, and z; Table 6 also summarizes hardware success.Source: §4.5 · pp. 8–9, Table 6 · p. 9

5 Conclusions and Limitations

WM-Craftnet uses a recurrent world model trained on multimodal observations and clean-depth targets to condition dexterous in-hand manipulation. The paper reports improved rotation and transfer, while noting limits in short-horizon tasks and severe-perturbation recovery.

5 Conclusions and Limitations

  • WM-Craftnet learns latent dynamics from noisy depth, proprioception, tactile sensing, and actions, reconstructs clean depth, and conditions the policy on its deterministic recurrent feature.Depth reconstruction is reported to suppress sensor artifacts while preserving hand-object geometry.Source: §5 · p. 9
  • The predictive representation serves as a reusable prior for a downstream policy trained on 49 objects.Source: §5 · p. 9
  • Experiments report improved multi-object rotation, unseen-object transfer, perturbation recovery, and sim-to-real performance over baseline visuotactile methods.Source: §5 · p. 9
  • Current tasks remain short-horizon, and recovery under severe perturbations remains limited.Failures occur when objects exceed the hand workspace, large offsets leave the recovery basin, or drift and jamming disrupt stable contacts.Source: §5 · p. 9

A Implementation Details

The implementation appendix provides details intended to support reproduction of WM-Craftnet, including its interfaces, training settings, observations, reward, and sim-to-real randomization.

A Implementation Details

A.1 World Model Training Details

WSM is trained from multimodal observations and actions, with reward used as a prediction target and clean depth supervising reconstruction. Its recurrent deterministic feature supplies policy context, while additional prediction heads and continued downstream adaptation support transfer to a new object distribution.

A.1 World Model Training Details

  • The world-model update receives proprioception, tactile/contact data, noisy depth, the previous action, and an episode-start flag; reward is only a prediction target.Source: §A.1 · p. 12
  • The RSSM updates at every control step, and only its detached deterministic feature, projected by an MLP, is exposed as actor context.Source: §A.1 · p. 12
  • Online PPO rollouts populate a replay buffer, and contiguous chunks train clean-depth and proprioception reconstruction, tactile-contact and reward prediction, and Dreamer-style dynamics and representation KL terms.Source: §A.1 · p. 12
  • The full WSM also predicts object pose, value, and object shape, with these heads used in pretraining and downstream adaptation.The shape target is a basis-point-set-to-mesh displacement vector.Source: §A.1 · p. 12
  • For 49-object downstream PPO, transferred WSM components continue adapting with new rollout data while the actor–critic and task-specific heads are learned for the target distribution.Source: §A.1 · p. 12

A.2 Policy Training Details

Policy execution smooths relative target updates for the 22 hand joints before hardware control. The reward balances goal-directed rotation with stability and contact-related objectives, and reset conditions limit uninformative exploration after failures.

A.2 Policy Training Details

  • The actor outputs relative target updates for all 22 hand joints, which a signal-level filter smooths before execution by the 10 Hz hardware controller.Source: §A.2 · p. 12
  • The reward favors stable, goal-directed, human-like rotation rather than speed alone.Source: §A.2 · p. 12
  • Reward terms penalize instability, drift, off-axis rotation, high torque, work, large actions, and tracking error, while contact and pose terms encourage useful fingertip proximity and natural postures.Episodes reset after drops, excessive axis deviation, prolonged insufficient target-axis spin, or reaching the horizon, reducing uninformative exploration after irrecoverable failures.Source: §A.2 · p. 12

A.3 System Identification and Domain Randomization

The real-robot transfer setup calibrates simulated joint dynamics against the hardware, then uses domain randomization to cover residual uncertainty. The policy keeps the same deployable observation/action interface in simulation and on the robot.

A.3 System Identification and Domain Randomization

  • The actor uses deployable signals, while simulator-only state supports reward, critic training, and diagnostics; implementation settings are summarized in Table 7.The same policy/control rate is used in simulation and on hardware, with simulation physics at 60 Hz.Source: §A.3 · p. 12, Table 8 · p. 13, Table 7 · p. 13
  • Post-calibration, simulated joint trajectories match real responses with an average sim-real tracking error below 0.2°.Calibration adjusts simulator Kp and Kd gains to align trajectory amplitude and phase after 1 Hz sinusoidal commands.Source: §A.3 · pp. 12–13
  • Training randomizes object mass, friction, gains, observation/action noise, reset pose, external forces, and gravity direction to cover uncertainty and encourage perturbation recovery.Object mass is sampled from 0.2 to 0.4 kg during training, with a wider test range.Source: §A.3 · p. 13

B Additional Experiments

The additional experiments cover controlled WSM ablations, large-scale object generalization, and tool-use-style manipulation.

B Additional Experiments

B.1 Additional WSM Mechanism Ablations

The additional experiments compare predictive WSM training with recurrent-history baselines and auxiliary-head variants, then examine large-scale object transfer, screwdriver manipulation, and supplementary rollouts.

B.1 Additional WSM Mechanism Ablations

  • Joint-level system identification yields average sim-real tracking error below 0.2° across evaluated joints.Source: Figure 7 · p. 14
  • 753.3 return and 1.293 RotR for Full WSM exceed LSTM’s 621.0 and 1.186 and GRU’s 497.8 and 1.095, while OffAxis 1.225 is comparable to GRU’s 1.226.Full WSM also lowers AngVar to 1.324; the comparison isolates predictive multimodal training from recurrent memory alone.Source: §B.1 · p. 14, Table 9 · p. 14
  • Combining all auxiliary heads achieves the strongest overall return and rotation rate; value prediction gives the largest single-head return gain, while pose prediction gives the lowest angular-velocity variation.Removing all auxiliary heads yields 688.1 return.Source: §B.1 · p. 14, Table 10 · p. 14

B.2 Large-Scale Object Set

  • After pretraining on nine z-axis objects, the WSM is transferred to a 49-object training set; recurrent states show coherent trajectories, partial object-dependent regions, and substantial overlap.The reported pattern is consistent with states reflecting both object-dependent geometry and interaction phases shared across objects, rather than strict object-identity classification.Source: §B.2 · p. 15, Figure 8 · p. 15

B.3 Screwdriver Tool-Use Manipulation

  • The same WM-Craftnet interface adjusts a screwdriver to a target position in goal-conditioned translation and rotates it while maintaining contact.These tasks probe tool-use-style behavior with elongated geometry, local handle contacts, and slip risk.Source: §B.3 · p. 15, Figure 9 · p. 15

B.4 Supplementary rollouts and failure cases.

  • The project website provides supplementary real-world rollouts, qualitative comparisons, and failure cases involving depth noise, slip, and difficult contact transitions.Source: §B.4 · p. 15

Related papers

Loading 2609.07002v1…