PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Versatile Humanoid Manipulation with Touch Dreaming

Yaru Niu, Zhenlong Fang, Binghong Chen, Shuai Zhou, Revanth Krishna Senthilkumaran, Hao Zhang, Bingqing Chen, Chen Qiu, H. Eric Tseng, Jonathan Francis, Ding Zhao

arXiv:2604.13015v3cs.RO

TL;DR

Humanoid loco-manipulation remains difficult because it must coordinate stability, dexterity, and changing contacts. The paper builds an integrated whole-body system and trains HTD with touch dreaming, achieving a 90.9% relative improvement in average success rate across five real-world tasks.

  • Problem

    Existing systems rarely combine stable whole-body execution, full dexterous-hand control, and tactile sensing for contact-rich humanoid manipulation.

  • Method

    The paper combines an RL lower-body controller, VR-based whole-body data collection, and single-stage HTD behavioral cloning augmented with future force and tactile-latent prediction.

  • Results

    90.9% relative improvement in average success rate over the stronger ACT baseline was achieved across five real-world contact-rich tasks.

  • Takeaways & Limitations

    The results support predictive touch dreaming as part of a practical system for dexterous humanoid manipulation under frequent contact changes.

Abstract

from arXiv · show

Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based lower-body controller that serves as the stability backbone for whole-body execution during complex manipulation. Building on this controller, we develop a VR-based whole-body humanoid data collection system that integrates dexterous hands and tactile sensing for contact-rich manipulation. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, with tactile-latent targets provided by an exponential moving average target encoder without requiring a separate tactile pretraining stage. This encourages the policy to learn contact-aware representations for dexterous manipulation. Across five real-world contact-rich tasks, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline for each task. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that our touch-dreaming-enhanced learning system enables versatile, high-dexterity humanoid manipulation in the real world. More information and open-source materials are available at humanoid-touch-dream.github.io.

I. INTRODUCTION

Real-world humanoid loco-manipulation requires coordinated whole-body stability, dexterous end-effectors, and contact-aware interaction. The paper addresses this gap with an integrated tactile-enabled system and HTD, which improves performance across five contact-rich tasks.

  • Whole-body humanoid loco-manipulation is challenging because frequent contact changes couple stability, torso posture, base motion, and end-effector dexterity.
  • Few existing systems jointly provide stable whole-body execution, full dexterous-hand control, and tactile sensing for real-world manipulation.
  • HTD treats touch as a core modality alongside multi-view vision and proprioception in a single-stage behavioral-cloning policy.
  • HTD predicts action chunks, future hand-joint forces, and future tactile latents supervised by an EMA target encoder without separate tactile pretraining.
  • 90.9% relative improvement in average success rate was achieved over the stronger ACT baseline across five real-world contact-rich tasks.
  • The framework develops a tactile-enabled whole-body humanoid manipulation system for stable, dexterous, contact-rich real-world manipulation.

B. Imitation Learning for Humanoid Manipulation

The paper combines whole-body teleoperation, synchronized multimodal demonstrations, and single-stage touch-aware imitation learning. Its central design uses future-touch prediction inside the policy rather than as a separate world model or inference-time module.

  • Related Work: Recent humanoid manipulation systems broaden data collection and learning paradigms across upper-body tasks and whole-body loco-manipulation.
  • Related Work: Most existing humanoid systems do not incorporate tactile sensing or explicitly model tactile signals in the learned policy.
  • Representation Learning for Contact-Rich Manipulation with Tactile Sensing: Predictive tactile learning has been used to improve contact-aware representations, planning, or reactive control.
  • Representation Learning for Contact-Rich Manipulation with Tactile Sensing: Future-touch prediction is an auxiliary objective inside a single-stage whole-body imitation policy, keeping training and deployment simple.
  • A System for Versatile Humanoid Dexterous Manipulation: The system uses an RL-based lower-body controller as the stable execution backbone for teleoperation and HTD deployment.
  • A System for Versatile Humanoid Dexterous Manipulation: VR teleoperation maps human head, wrist, and hand motions into torso commands, end-effector pose targets, and dexterous hand targets, with joystick-based base velocity.
  • A System for Versatile Humanoid Dexterous Manipulation: The collected dataset synchronizes camera views, proprioception, hand-force signals, tactile observations, and whole-body action targets.
  • Representation Learning for Contact-Rich Manipulation with Tactile Sensing: HTD tokenizes multimodal observations into a shared representation and predicts structured body-and-hand actions plus future hand forces and tactile latents.

B. Lower-body Controller

The lower-body controller provides deployable, command-conditioned joint targets for tracking base motion and torso pose, forming a stability-oriented foundation for whole-body execution.

  • B. Lower-body Controller: The lower-body policy is command-conditioned to track base motion and torso pose targets.Its observations include body-frame base angular velocity, projected gravity, lower-body joint states, and the previous lower-body action.
  • B. Lower-body Controller: 15-dimensional lower-body actions specify target joint positions for two six-joint legs and three waist motors.The output is q_lower ∈ R15.
  • B. Lower-body Controller: The teacher is trained in simulation with PPO using privileged information, then distilled into a real-world-deployable student through DAgger.The student uses only real-world-available observations, while the teacher has access to simulated foot-contact indicators.
  • B. Lower-body Controller: A two-timestep proprioceptive history is concatenated to compensate for partial observability during student-policy training.The student rolls out its own actions and minimizes an L2 loss against the teacher’s reference actions.
  • B. Lower-body Controller: Observation noise and randomized physics parameters are applied during training to improve sim-to-real transferability.Command signals are sampled from predefined ranges covering base velocity, yaw rate, torso orientation, and torso height.

C. Teleoperation and Data Collection

The system combines VR motion mapping, lower-body stabilization, inverse-kinematics execution, and dexterous hand retargeting to collect synchronized multimodal humanoid demonstrations for HTD.

  • C. Teleoperation and Data Collection: VR motion mapping converts operator head, wrist, and hand motions into unified robot-frame torso, wrist, hand, and base-velocity targets.These targets support synchronized whole-body command execution during real-world data collection.
  • C. Teleoperation and Data Collection: The execution stack uses the lower-body controller for stable locomotion and torso tracking, inverse kinematics for wrist poses, and DexPilot-based retargeting for hand joints.DexPilot optimizes fingertip-distance consistency for grasping and in-hand interaction.
  • C. Teleoperation and Data Collection: Teleoperation records RGB views, robot and hand proprioception, per-joint hand forces, and tactile readings from both hands.Each hand provides a 1062-dimensional tactile observation distributed across 17 spatial sensing regions.
  • D. Learning Dexterous Manipulation with Touch Dreaming: HTD tokenizers encode each observation modality into fixed-length tokens using modality-specific features and cross-attention aggregation.Image streams use separate tokenizers for the head camera and wrist cameras, while tactile inputs are organized per finger or hand region.
  • D. Learning Dexterous Manipulation with Touch Dreaming: An encoder–decoder Transformer fuses observation tokens and emits fixed output tokens consumed by action and dream experts.Action experts predict pose, velocity, and hand-action targets, while dream experts predict future forces and tactile latents.
  • D. Learning Dexterous Manipulation with Touch Dreaming: Touch dreaming predicts future hand-joint forces and tactile latents in learned latent space using stable EMA-teacher supervision.The auxiliary predictions regularize the shared trunk toward contact-aware representations, avoiding direct raw-tactile regression.

E. Training Paradigm

HTD trains with single-stage behavioral cloning augmented by touch dreaming, predicting action chunks alongside future hand forces and tactile latents. An EMA teacher supplies stable latent targets, while force and tactile objectives are combined with behavior cloning.

  • Touch Dreaming: Touch dreaming adds future hand-joint force and tactile-latent prediction to action-chunk behavioral cloning.Force prediction uses smooth L1 regression; tactile prediction operates on future latent targets rather than raw tactile arrays.
  • EMA Teacher: The tactile targets are generated by an EMA teacher encoder, which evolves slowly and receives no gradient updates.The stop-gradient EMA target prevents coadaptation toward near-constant latent representations.
  • Training Objective: The overall objective combines behavior-cloning, future force-prediction, and tactile-latent-prediction losses weighted by λF and λZ.The action, force, and latent objectives use smooth L1-based losses, with tactile prediction combining cosine direction and magnitude alignment terms.
  • Latent Supervision: Latent tactile supervision predicts compact representations instead of raw tactile heatmaps.The tactile loss uses cosine similarity together with magnitude alignment to avoid unit-norm collapse.

IV. EXPERIMENTS

The experiments examine lower-body control, real-world humanoid manipulation, predictive touch dreaming, and whether touch dreaming captures contact-aware representations.

  • Research Questions: The experiments address four questions spanning LBC tracking, HTD manipulation performance, tactile-prediction ablations, and contact-aware representation quality.The questions compare LBC with other methods, evaluate five real-world tasks, test predictive touch dreaming and latent supervision, and assess learned representations.

A. How does our LBC strategy compare to other methods

The LBC is benchmarked against FALCON and AMO using tracking errors and stable controllable workspace measurements. It achieves the best overall tracking on most metrics while retaining a broad stable operating region.

  • Baselines: LBC is compared with FALCON and AMO, which use decoupled or hierarchical whole-body control strategies.The benchmark evaluates representative learned controllers with differing upper- and lower-body coordination mechanisms.
  • Evaluation Protocol: Tracking errors are averaged across 4096 parallel simulation environments over 500 timesteps.Metrics include linear and angular velocity, height, yaw, pitch, and roll tracking errors.
  • Tracking Results: LBC achieves the best overall tracking on most metrics, notably linear velocity, torso height, and torso orientation.AMO has slightly lower yaw-rate error, but LBC provides a better balance between locomotion tracking and whole-body configuration control.
  • Robustness: Height and especially pitch tracking show standard deviations exceeding their means under a small number of conflicting command combinations.Large forward torso pitch combined with low torso height can cause temporary tracking trade-offs for stability.
  • Stable Workspace: The stable workspace covers the trained height range and most of the yaw range, while roll is the most restrictive dimension.Boundary postures show stable crouching, bending, and large torso reorientation beyond nominal upright motions.

B. How do our learning system and HTD perform on real-

The learning system is evaluated on five real-world tasks covering precise insertion, rigid and deformable-object handling, tool use, and bimanual loco-manipulation. HTD consistently outperforms both ACT baselines across success rate and task score.

  • Tasks: The evaluation covers Insert-T, Book Organization, Towel Folding, Cat Litter Scooping, and Tea Serving.Together, the tasks test tight-tolerance insertion, limited-grasp rigid objects, deformable handling, tool-mediated interaction, and dual-arm loco-manipulation.
  • Training Data: The task-specific training sets contain 193, 150, 141, 153, and 72 trajectories for the five tasks, respectively.The demonstrations correspond to approximately one hour per task, except Tea Serving with about 40 minutes.
  • Ablations: Latent tactile prediction outperforms raw tactile prediction in the HTD ablations.The ablation compares variants without touch and dreaming, without dreaming, raw tactile dreaming, and latent tactile dreaming.
  • Main Results: HTD consistently outperforms ACT (Visual + Proprio) and ACT (Visual + Proprio + Touch) across all five tasks in success rate and task score.Each method is evaluated with 20 real-world trials per task using strict success rate and partial-progress task score.
  • Main Results: 30.0 percentage points and 17.9 percentage points are HTD’s improvements in task-averaged success rate and task score over the better ACT baseline.These correspond to relative gains of about 90.9% and 31.1%, respectively.
  • Analysis: The largest HTD gains occur on tasks demanding contact-aware control or whole-body coordination.The reported overall results indicate that the full HTD framework is better suited than ACT baselines for versatile humanoid loco-manipulation.

C. How do predictive touch dreaming and latent tactile su-

HTD’s touch-dreaming ablations show that predicting future tactile latents is more effective than predicting raw tactile signals. Qualitative rollouts indicate that dreamed forces and latents track contact dynamics while remaining stable during sustained contact.

  • Ablation Results: 30% relative gain in average success rate makes Dream Latent Tactile outperform Dream Raw Tactile.Dream Latent Tactile achieves the best overall performance in average metrics.
  • Qualitative Analysis: Dreamed future hand forces track the timing and magnitude of contact events during Tea Serving and Towel Folding rollouts.Predicted tactile heatmaps are close to ground-truth heatmaps at most timesteps, with temporary similarity drops during abrupt contact transitions.
  • Qualitative Analysis: Dreamed tactile latents remain relatively stable during sustained contact, while localized similarity drops coincide with sudden force spikes.The deviations are attributed to rolled-out latent chunks and noisy variation in raw tactile readings.
  • Representation Quality: Dreamed latents are more stable and semantically aligned with contact changes than ground-truth latents that vary with noisy or spatially sparse tactile readings.The representations are inferred jointly from vision and applied forces rather than directly matching raw tactile readings.
  • Contact Awareness: Light or sparse contacts produce consistent latent patterns, whereas richer contacts activate distinct, high-intensity patterns across tasks and fingers.Tea Serving generally involves larger applied forces than Towel Folding because it uses rigid objects.
  • Overall Results: Across five real-world tasks, HTD improves average success rate by 90.9% relative to the stronger ACT variant for each task.The evaluated tasks cover insertion, rigid-object manipulation, deformable-object handling, tool use, and bimanual loco-manipulation.

A. Lower-Body Controller Details

The lower-body controller is trained with randomized commands and physics conditions to support sim-to-real control. Its stable ranges cover the trained height range, most yaw range, and broad but asymmetric pitch range, while roll remains more restrictive.

  • Training Setup: Command signals are uniformly sampled over predefined ranges for planar base velocity, yaw rate, torso orientation, and torso height.The training commands include vxy, ωz, torso roll-pitch-yaw, and torso height h.
  • Training Setup: Observation noise and randomized physics parameters are applied to improve sim-to-real transferability.Domain randomization accompanies the predefined command ranges during controller training.
  • Achievable Control Ranges: The learned policy achieves stable torso-height control over [0.33, 0.80] m, roll over [−0.38, 0.35] rad, pitch over [−0.92, 1.41] rad, and yaw over [−1.50, 1.34] rad.These are the maximum stable controllable ranges summarized in Table V.
  • Achievable Control Ranges: The controller fully covers the trained height range and slightly exceeds its lower bound, while covering most of the trained yaw range.The stable pitch range extends beyond training on the negative side but is slightly smaller near the positive extreme.
  • Achievable Control Ranges: The narrower roll range indicates that lateral whole-body balance remains the most restrictive direction.Pitch is broad but asymmetric, whereas roll is noticeably narrower than the training range.

B. Reward Details

The lower-body controller’s reward combines tracking, regularization, contact and gait, stability, and auxiliary terms. Together, these terms target commanded motion, smooth actuation, stable foot-ground interaction, balanced posture, and safe joint behavior.

  • Tracking Terms: Tracking rewards encourage commanded planar velocity, yaw rate, torso height, torso roll, torso pitch, and torso yaw.Torso orientation terms support posture tracking and independent upper-body reorientation for manipulation.
  • Regularization Terms: Regularization penalizes energy use, rapid action changes, large joint accelerations, undesired vertical motion, and excessive roll-pitch rates.These terms promote smoother, less abrupt, and more physically plausible transitions while improving upper-body stability.
  • Contact and Gait Terms: Contact and gait terms discourage non-foot collisions, foot slip, airborne states, excessive foot forces, and abnormal stumbling patterns.They also reward sustained single-stance phases and bilateral standing contact under near-zero motion commands.
  • Safety and Auxiliary Terms: Joint torque-limit penalties use normalized thresholds of 0.9 for the waist and 0.99 for ankle and hip-pitch joints.The penalties target computed torques exceeding group-specific fractions of actuator limits.
  • Stability Terms: Stability terms penalize soft joint-limit violations, non-flat orientation, overly close feet, and poor support geometry.Additional rewards align the center of gravity with the ankle midpoint and maintain a 0.27 m knee stance width.
  • Safety and Auxiliary Terms: A termination penalty strongly discourages catastrophic failure such as falling or unrecoverable instability.The reward design also includes joint-deviation penalties that discourage departures from selected default configurations.
Loading 2604.13015v3…