PaperlayerView exact revision on arXiv

Source-linked AI summary

Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation

Yunfan Lou, Yifan Ye, Yankai Fu, Jun Cen, Xiaowei Chi, Yaoxu Lyu, Peidong Jia, Sirui Han, Zhihe Lu, Shanghang Zhang

arXiv:2606.08737v1cs.RO

TL;DR

Vision-dominant world action models lack important physical-interaction cues in contact-rich manipulation. Dream-Tac jointly models actions and future visual-tactile states with contact-aware fusion and attention, achieving stronger task performance while adding training and inference acceleration.

  • Problem

    Vision-only world action models struggle in contact-rich manipulation because tactile interaction cues are ambiguous or unavailable in RGB observations.

  • Method

    Dream-Tac jointly predicts future visual observations, future tactile observations, and robot actions using contact-aware tactile integration, attention bias, and dual-level acceleration.

  • Results

    Dream-Tac consistently outperforms strong baselines across six real-world contact-rich tasks, achieving higher success rates and more accurate future visual predictions.

  • Takeaways & Limitations

    Unified visuo-tactile world modeling improves tactile utilization and supports more practical real-time contact-rich manipulation.

  • Takeaways & Limitations

    Evaluation covers a limited set of tasks, while generalization to broader task families, diverse objects, and more complex environments remains unverified.

Abstract

from arXiv · show

World action models inherit the predictive capability of world models, enabling action generation to be guided by anticipated future observations. However, they rely primarily on vision and often fail in contact-rich manipulation, where critical cues arise from physical interaction. In this paper, we propose Dream-Tac, a unified Tactile-World Action Model that jointly models actions, future visual observations, and tactile dynamics. Specifically, Dream-Tac introduces (i) contact-gated visuotactile fusion to selectively integrate tactile signals and (ii) a contact-aware attention bias to better regulate cross-modal interactions during manipulation. To support real-time deployment, we further design a dual-level acceleration strategy, reformulating the contact-aware bias to preserve the fused attention path during training and introducing cache-based diffusion acceleration at inference, achieving up to 2.9$\times$ faster training and 1.8$\times$ faster inference. Across six contact-rich manipulation tasks, Dream-Tac improves action accuracy by 31.7\% on average, demonstrating the effectiveness of unified visuotactile world modeling.Code is available at https://github.com/LYFCLOUDFAN/Dream-Tac.

1 Introduction

Dream-Tac addresses the limits of vision-only world action models in contact-rich manipulation by jointly modeling tactile signals, visual futures, and actions. It also adds contact-aware attention and acceleration strategies for more effective and practical deployment.

  • Vision-only world action models struggle with contact-rich manipulation because RGB observations miss contact states, local geometry, and fine-grained object properties.
  • Tactile signals provide direct contact and local interaction cues, but their sparse, transient nature makes uniform token treatment prone to diluting critical events.
  • Dream-Tac jointly predicts future visual observations, robot actions, and future tactile observations within a generative world action model.
  • Contact-aware attention adaptively amplifies tactile influence during salient contact dynamics instead of treating tactile tokens uniformly.
  • Experiments on contact-rich manipulation tasks report the highest success rate among four strong baselines and 31.6% improvement over Cosmos Policy.
  • 2.94× training speedup and 1.8× inference speedup result from FlashAttention-based gated bias and diffusion-step caching, respectively.

2 Related Work

Prior work shows tactile sensing improves contact-aware control and representation learning, but future contact-dynamics modeling remains underexplored. Dream-Tac extends joint world action modeling beyond vision by predicting future visual and tactile states together with actions.

  • Tactile sensing has been applied to grasping, insertion, in-hand manipulation, and tool-use tasks to capture contact and interaction cues.
  • Visuo-tactile policies combine tactile feedback with diffusion control, 3D features, force-guided attention, or unified vision-language-action modeling.
  • Tactile representation-learning and pretraining methods seek transferable contact-aware features for downstream manipulation.
  • Explicitly modeling future tactile-state evolution and contact dynamics remains underexplored in existing tactile research.
  • Dream-Tac incorporates tactile sensing into joint world action modeling, predicting future visual observations, future tactile signals, and robot actions.

3 Method

Dream-Tac formulates manipulation as unified prediction of actions and future visual-tactile states, using a shared latent DiT backbone with contact-aware attention. A data-derived gate selectively increases tactile access during interaction events, while the denoising objective jointly supervises all predicted modalities.

  • 3.1 Problem Formulation: The task is to predict an H-step action chunk from current visual and tactile observations plus a language instruction.
  • 3.1 Problem Formulation: World action modeling combines action prediction with future visual prediction so anticipated scene evolution provides structure for action generation.
  • 3.1 Problem Formulation: Dream-Tac extends this formulation by conditioning on tactile observations and predicting future tactile observations alongside future visuals and actions.
  • 3.2 Dream-Tac Architecture: A pretrained video DiT, T5 encoder, and video VAE place visual observations, robot states, actions, and tactile observations into a shared latent modeling framework.
  • 3.2 Dream-Tac Architecture: Bidirectional self-attention lets action tokens condition on future visual and tactile changes, supporting anticipated contact dynamics for precise manipulation.
  • 3.3 Contact-Aware Self Attention: A gated additive attention bias selectively increases non-tactile queries' access to tactile keys while leaving tactile-to-tactile interactions unchanged.
  • 3.3 Contact-Aware Self Attention: The contact gate is computed from consecutive tactile-frame changes and increases with salient change in either fingertip view.
  • 3.4 Training Objective: The unified latent denoising objective simultaneously supervises action generation, future visual prediction, and future tactile prediction.

4 Experiment

Dream-Tac is evaluated on six real-world contact-rich manipulation tasks for effectiveness, efficiency, generalization, and ablation, with strong average success and targeted gains under tactile-relevant conditions. Experiments also examine tactile fusion, contact-aware attention, acceleration, shared tactile representations, and gate behavior.

  • Performance on Real-World Experiments: 83.3% average success rate makes Dream-Tac the best overall method across six real-world tasks, ahead of Cosmos-Policy, ForceVLA, π0.5, and π0.The corresponding baselines achieve 51.7%, 50.8%, 45.0%, and 30.8%, respectively.
  • Performance on Real-World Experiments: Dream-Tac achieves the highest success rate on five of six tasks, including 100% on Pick Baguette and Play Mahjong.It also reaches 35% on Insert USB, 90% on Clean Whiteboard, and 90% on Cut Banana; on Peel Cucumber, it achieves 85% and remains slightly below ForceVLA.
  • Performance on Real-World Experiments: Dream-Tac’s largest advantages occur on precise contact-rich tasks such as Insert USB and Cut Banana, whereas coarse tasks such as Pick Baguette are often solvable through vision and basic motion control.The paper links the harder tasks to subtle contact transitions, spatial alignment, and forceful interaction.
  • Ablation Studies: Tactile fusion raises average success from 51.7% to 74.2%, while adding contact-aware attention bias further increases it to 83.3% across six tasks.The ablation compares Visual WAM, Visuo-tactile WAM, and Visuo-tactile WAM + Bias.
  • Generalization Under Environment Variations: Dream-Tac outperforms Cosmos-Policy in three of four environment-variation settings and matches it on spatial generalization.It improves table-height, object, and background generalization, while both methods achieve 100% in-distribution and 80% under unseen placements for Pick Baguette.
  • Training and Inference Efficiency: 66.0% training-time reduction and 1.8× inference speedup improve Dream-Tac’s efficiency when tactile input, attention bias, and timestep caching are used.In the full training setting, time drops from 80.82 s to 27.48 s; caching preserves 85% success while reducing latency from 1109 ms to 619 ms.
  • Tactile Representation Analysis: The pretrained Wan VAE separates tactile patterns into distinct latent clusters despite limited pixel-level visual differences.This indicates that the shared latent space preserves discriminative tactile cues useful for manipulation.
  • Contact Gate Behavior: Peaks in tactile change ρ_t coincide with increases in gate g_t, while flatter segments keep the gate lower and contact stages sustain higher gating.Pre-contact fluctuations remain relatively low, and an episode beginning near object contact activates the gate earlier.

5 Conclusion

Dream-Tac is a unified tactile world action model for contact-rich manipulation that jointly models actions, future visual observations, and tactile dynamics. Across six real-world tasks, it consistently outperforms strong baselines, with higher success rates and more accurate future visual predictions.

  • Dream-Tac jointly models robot actions, future visual observations, and tactile dynamics for contact-rich manipulation.
  • Contact-gated visuotactile fusion and contact-aware attention bias selectively emphasize tactile information during important contact transitions.
  • Dual-level acceleration makes Dream-Tac more practical for real-time deployment.
  • Across six real-world contact-rich manipulation tasks, Dream-Tac consistently outperforms strong baselines with higher success rates and more accurate future visual predictions.

A.1 Real-World Experimental Setup

The real-world platform combines a Franka Emika Panda robot, fixed and wrist-mounted RealSense D435i cameras, and fingertip tactile sensors. The setup provides complementary global, local, and contact observations for manipulation tasks involving sustained or precise physical interaction.

  • The experimental platform uses a Franka Emika Panda robot, two RealSense D435i cameras, and two Xense Photon tactile sensors on the gripper fingertips.
  • The fixed camera captures global scene layout, while the wrist-mounted camera provides local views near the interaction region.
  • Cucumber peeling requires continuous blade contact and moderate force while moving longitudinally along the curved surface.
  • USB insertion requires directional alignment, stable contact, and motion adjustments for smooth engagement.

A.3 Experimental Settings

The experiments use diverse teleoperated demonstrations with synchronized visual, tactile, and proprioceptive recordings, followed by randomized real-world evaluation against four baselines. Each task is evaluated under a common 20-trial protocol with bounded execution steps.

  • Each task uses 100 demonstration trajectories collected with randomized initial states and interactions.
  • Demonstrations record first-person and third-person camera observations, tactile images, and proprioceptive states at 30 Hz.
  • The evaluation settings include task-specific episode lengths, teleoperation times, and maximum execution steps.
  • Evaluation executes each task for 20 trials with randomized object positions, and trials fail when completion exceeds the maximum allowed steps.
  • Dream-Tac is compared with π0, π0.5, ForceVLA, and Cosmos Policy.

A.5 Training Hyperparameters

Dream-Tac is fine-tuned from a pretrained video-generation checkpoint using a hybrid denoising objective and synchronized visual-tactile-action modeling. Its contact gate derives from tactile frame changes, while caching and structured attention support efficient operation.

  • Dream-Tac is fine-tuned from the Cosmos-Predict2-2B Video2World checkpoint with Fused Adam and mixed bfloat16 precision.
  • The inherited rectified-flow / hybrid-EDM objective jointly noises and denoises action and future-prediction tokens while keeping observed context latents clean.
  • Synchronized visual, proprioceptive, action, and tactile inputs are used, with future tactile latents jointly denoised alongside future visual latents and action chunks.
  • The contact gate is computed directly from tactile RGB frame differences, without a learned gating network.
  • Most tactile changes are small and keep the gate low, while larger changes increase the gate rapidly after exceeding the typical noise range.

B.1 FlashBias Implementation

Dream-Tac accelerates attention and diffusion inference by preserving fused attention while caching redundant denoising computations. The resulting design reduces computational overhead without explicitly materializing dense bias matrices.

  • FlashBias Reformulation: Rank-one tactile bias is folded into dot-product attention by augmenting queries and keys with one scalar channel.The resulting inner product exactly adds the structured bias to the original attention logits.
  • FlashBias Reformulation: Fused attention avoids explicit O(S^2) bias materialization while adding only O(1) per-token state per head.Zero-padding can satisfy kernel alignment without changing the logits.
  • Training Acceleration: FlashBias substantially improves Dream-Tac training efficiency by preserving the optimized fused attention kernel.
  • Diffusion-Step Cache: Diffusion-step caching reduces repeated computation across similar denoising steps with limited approximation error.
  • Diffusion-Step Cache: 0.997 average cosine similarity between neighboring diffusion steps motivates cache-based inference acceleration.The high similarity indicates marginal output changes across most denoising steps.
  • Diffusion-Step Cache: Full forward computation is performed at the first and third steps, while cached results are reused for remaining steps.This preserves generation quality while reducing inference cost.

C Additional Reconstruction Results

Additional reconstruction experiments evaluate Dream-Tac's visual and joint visuo-tactile predictive representations. These results complement downstream manipulation evaluations by examining temporal coherence across modalities.

  • Dream-Tac is evaluated through both visual reconstruction and joint visuo-tactile reconstruction.The experiments assess predictive representations in multiple modalities.
  • The reconstructions show temporally coherent and physically meaningful future representations across modalities.These results complement the model's downstream manipulation performance.

C.1 Visual Reconstruction

Dream-Tac reconstructs future visual and tactile states that track manipulation progression and contact dynamics. Its tactile predictions distinguish pre-contact from sustained interaction while preserving the main contact pattern.

  • Visual Reconstruction: Dream-Tac faithfully captures global scene layout, robot-object geometry, and coarse motion trends in future visual frames.Predicted frames preserve spatial relationships among the gripper, knife, and banana.
  • Visual Reconstruction: Predicted visual frames remain semantically consistent with ground truth despite smoothed fine-grained textures and high-frequency details.The reconstructions reflect key manipulation-process progression.
  • Joint Visuo-Tactile Reconstruction: Joint visual-tactile reconstruction encourages future representations that capture contact-rich state transitions weakly observable in RGB images.Tactile signals provide clearer evidence of these transitions.
  • Tactile Reconstruction: During Pre-contact, predicted tactile images preserve the undeformed background pattern without spurious contact signals.
  • Tactile Reconstruction: During Cutting, predictions capture the main spatial structure and phase-dependent change of the contact pattern.Fine-grained pin-level details are slightly smoothed.
  • Tactile Reconstruction: Dream-Tac distinguishes quiescent from contact-active tactile states and predicts tactile feedback evolution alongside manipulation.This supports modeling touch as a cross-modal dynamic rather than an auxiliary signal.

D Limitations and Future Work

Dream-Tac is evaluated on a limited range of real-world contact-rich tasks and uses a relatively simple tactile gating mechanism. Future work targets broader data, richer contact modeling, and more challenging manipulation settings.

  • Limitations: Generalization beyond the evaluated real-world contact-rich tasks, objects, and environments remains unverified.
  • Limitations: The current gating mechanism may not capture subtle or long-horizon interaction patterns because it relies on simple frame-to-frame tactile variation.
  • Limitations: Diffusion-based world action models remain computationally expensive compared with lightweight reactive policies despite efficiency optimizations.
  • Future Work: Future work will scale Dream-Tac to larger visuo-tactile datasets and explore more expressive contact modeling mechanisms.
  • Future Work: Planned extensions include dexterous multi-stage manipulation, deformable-object interaction, and long-horizon contact-rich tasks.
Loading 2606.08737v1…