PaperlayerView exact revision on arXiv

Source-linked AI summary

Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation

Yunfan Lou, Yifan Ye, Yankai Fu, Jun Cen, Xiaowei Chi, Yaoxu Lyu, Peidong Jia, Sirui Han, Zhihe Lu, Shanghang Zhang

arXiv:2606.08737v1cs.RO

TL;DR

World action models rely mainly on vision, which may not capture cues needed for contact-rich manipulation. Dream-Tac jointly models actions, future visual observations, and tactile dynamics with contact-aware tactile integration; across six real-world tasks, it consistently outperforms strong baselines in success rates and future visual prediction accuracy.

  • Problem

    World action models rely mainly on vision, while contact-rich fine-grained manipulation depends on physical-interaction cues that RGB observations may not clearly reveal.

    Source: §1 · p. 2

  • Method

    Dream-Tac jointly predicts actions, future visual observations, and tactile signals, using contact-aware attention to emphasize tactile cues during salient interactions.

    Source: §1 · p. 2

  • Results

    Across six real-world contact-rich tasks, Dream-Tac consistently outperforms strong baselines; adding contact-aware attention raises average success rate from 74.2% to 83.3%.

    Source: §4.2.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    The experiments support the effectiveness of unified visuo-tactile world modeling for real-world contact-rich manipulation.

    Source: §5 · p. 9

    Evaluation is limited to a set of real-world contact-rich tasks, so generalization to broader task families, more diverse objects, and more complex environments remains unverified.

    Source: D Limitations and Future Work · p. 15

Abstract

from arXiv · show

World action models inherit the predictive capability of world models, enabling action generation to be guided by anticipated future observations. However, they rely primarily on vision and often fail in contact-rich manipulation, where critical cues arise from physical interaction. In this paper, we propose Dream-Tac, a unified Tactile-World Action Model that jointly models actions, future visual observations, and tactile dynamics. Specifically, Dream-Tac introduces (i) contact-gated visuotactile fusion to selectively integrate tactile signals and (ii) a contact-aware attention bias to better regulate cross-modal interactions during manipulation. To support real-time deployment, we further design a dual-level acceleration strategy, reformulating the contact-aware bias to preserve the fused attention path during training and introducing cache-based diffusion acceleration at inference, achieving up to 2.9$\times$ faster training and 1.8$\times$ faster inference. Across six contact-rich manipulation tasks, Dream-Tac improves action accuracy by 31.7\% on average, demonstrating the effectiveness of unified visuotactile world modeling.Code is available at https://github.com/LYFCLOUDFAN/Dream-Tac.

1 Introduction

Dream-Tac extends world action modeling to contact-rich manipulation by jointly predicting future visual observations, tactile signals, and robot actions. It uses contact-aware attention to emphasize tactile cues during salient interactions and adds training and inference acceleration.

Motivation

  • Vision alone often misses contact states, local geometry, and fine-grained object properties needed for precise control in contact-rich manipulation.Tactile sensing provides contact and local-interaction signals that may be ambiguous or unavailable in RGB observations.Source: §1 · p. 2
  • Because tactile signals are sparse and transient, treating their tokens symmetrically with other modalities risks diluting cues such as contact onset, slip, and release.Their event-driven nature motivates selectively incorporating tactile signals into world action models.Source: §1 · p. 2

Contributions

  • Dream-Tac achieved the highest success rate among four strong baselines and outperformed Cosmos Policy by 31.6%.The introduction reports real-world experiments validating the model's effectiveness.Source: §1 · p. 2
  • Dream-Tac jointly predicts future visual observations, tactile signals, and robot actions, using contact-aware attention to amplify touch when contact dynamics become salient.The model builds on a pretrained video generative backbone and prioritizes interaction-relevant tactile signals rather than treating tactile tokens uniformly.Source: §1 · p. 2
  • A dual-level acceleration system uses FlashAttention-based gated-bias attention for training and diffusion-step caching for inference, achieving up to 2.94× and 1.8× speedups, respectively.The optimizations address the added computational burden of tactile sequences and iterative diffusion denoising.Source: §1 · p. 2

2 Related Work

Prior tactile and visuo-tactile policies support contact-rich manipulation, but explicitly modeling future contact dynamics remains underexplored. Dream-Tac extends world action modeling to jointly predict visual observations, tactile signals, and actions.

2.1 Tactile in Robot Manipulation

  • Tactile signals capture interaction cues in grasping [8], insertion [15], in-hand manipulation [38], and tool use [3] [41].Visuo-tactile perception has also been incorporated into end-to-end manipulation policies [6] [20] [50] [52].Source: §2.1 · p. 2
  • Prior visuo-tactile policies including RDP [45], AdapTac [27], and Tactile-VLA [24], plus pretraining [22] [44] [55], advance control, but future contact-dynamics modeling remains underexplored.RDP combines slow diffusion with fast tactile feedback, while AdapTac uses force-guided attention and Tactile-VLA incorporates tactile sensing into a unified VLA framework.Source: §2.1 · pp. 2–3

2.2 World Action Model

  • Dream-Tac predicts future visual observations, tactile signals, and robot actions jointly, extending world action modeling beyond vision to capture contact dynamics.The model conditions future visual and tactile evolution on robot actions, supporting predictive contact-aware manipulation.Source: §2.1 · p. 3, §2.2 · p. 3
  • World action models use future visual prediction to guide action generation through imagine-then-execute [23] [28] [53] or joint modeling [26] [46] [54].Joint modeling methods learn actions and future observations within a unified generative process.Source: §2.2 · p. 3

3 Method

Dream-Tac extends world action modeling by jointly predicting robot actions, future visual observations, and future tactile observations from visual, tactile, and language inputs. Its shared diffusion-transformer design uses contact-gated attention, a unified denoising objective, and acceleration strategies for deployment.

3.1 Problem Formulation

  • Dream-Tac jointly predicts action chunks and future visual and tactile observations from current visual and tactile inputs and a language instruction.This extends world action modeling beyond future visual prediction to include future contact dynamics.Source: §3.1 · p. 3

3.2 Dream-Tac Architecture

  • A pretrained video DiT processes visual, tactile, robot-state, action, and language tokens in a shared latent space, enabling bidirectional attention across future modalities.The architecture reuses the pretrained T5 encoder and video VAE; the VAE’s latent space separates tactile patterns associated with different actions without additional tactile-specific pretraining.Source: §3.2 · p. 3

3.3 Contact-Aware Self Attention

  • Tactile frame changes set a per-timestep gate that strengthens an asymmetric attention bias from non-tactile queries to tactile keys during salient contact events.The gate uses raw tactile RGB differences and fixed robust normalization, requiring no learned gating network or privileged future information.Source: §3.3 · pp. 4–5

3.4 Training Objective

  • A shared latent denoising objective jointly supervises action generation, future visual prediction, and future tactile prediction.The loss can be decomposed into action, image, and tactile denoising terms, with weights balancing the visual and tactile objectives.Source: §3.4 · p. 5

3.5 Efficiency Design

  • Training latency falls from 97 ms to 29 ms per iteration on NVIDIA H200 GPUs, while diffusion-step caching cuts effective inference steps from 10 to 2.The gated-bias attention uses a low-rank formulation on FlashAttention; with 10 denoising steps, the method runs at 5Hz on an A800 GPU before caching.Source: §3.5 · p. 5

4 Experiment

Dream-Tac is evaluated on six real-world contact-rich manipulation tasks, with results showing stronger average performance, generalization, and efficiency, alongside evidence that tactile representations and contact gating support the method.

4.2 Evaluations

  • Dream-Tac achieves an 83.3% average success rate, outperforming Cosmos-Policy (51.7%), ForceVLA (50.8%), π0.5 (45.0%), and π0 (30.8%).The evaluation covers six real-world tasks; Figure 3 accompanies the overall comparison.Source: §4.2.1 · p. 6, Figure 3 · p. 7

4.2.1 Performance on Real-World Experiments.

  • Dream-Tac leads on five of six tasks, excels on fine-contact tasks, and remains robust when Mahjong vision is fully occluded.It reaches 100% on Pick Baguette and Play Mahjong, 35% on Insert USB, 90% on Clean Whiteboard and Cut Banana, and 85% on Peel Cucumber—slightly below ForceVLA. Under full Mahjong occlusion, its tactile attention bias supports more tactile-grounded action than ForceVLA.Source: §4.2.1 · p. 6

4.2.2 Ablation Studies.

  • Adding tactile observations raises average success from 51.7% to 74.2%, and adding contact-aware attention bias further raises it to 83.3%.The ablation compares visual-only WAM, visuo-tactile WAM without bias, and visuo-tactile WAM with bias, attributing the main gain to tactile fusion and an additional gain to the bias.Source: §4.2.2 · p. 6, Table 1 · p. 6

4.2.3 Generalization Under Environment Variations.

  • Dream-Tac outperforms Cosmos-Policy in three of four environment variations and matches it on spatial generalization.It achieves 85%, 90%, and 75% under standard, +5 cm, and -5 cm table-height settings; Mahjong success is 35% on training tiles and 100% on unseen appearances, while Cut Banana reaches 90% and 70% under standard and altered backgrounds versus Cosmos-Policy’s 40% and 25%. Both methods score 100% in-distribution and 80% on unseen placements for Pick Baguette. Its advantage is strongest for variations affecting tactile-relevant cues.Source: §4.2.3 · pp. 6–7, Figure 4 · p. 8

4.3 Training and Inference Efficiency

  • In the full setting, Dream-Tac reduces training time from 80.82 s to 27.48 s, a 66.0% reduction and 2.9× speedup over Cosmos-Policy.Across settings, the speedups are 1.8× without tactile input and bias and 1.5× with tactile input but no bias. A low-rank bias reformulation integrated into blockwise attention preserves the fused attention path and reduces HBM accesses; the structured bias, rather than tactile tokens, is the main computational bottleneck.Source: §4.3 · p. 7, Figure 5 · p. 8
  • Timestep caching maintains an 85% success rate while reducing inference latency from 1109 ms to 619 ms, a 1.8× speedup.This compares the accelerated system with full 10-step denoising for real-time control.Source: §4.3 · p. 7, Figure 5 · p. 8

4.4.1 Latent Separability of Tactile Representations.

  • The pretrained Wan VAE separates tactile patterns into distinct shared-latent clusters despite limited pixel-level visual differences.The finding suggests that projecting tactile observations with RGB preserves discriminative tactile cues useful for manipulation.Source: §4.4.1 · p. 7, Figure 6 · p. 8

4.4.2 Contact gate behavior.

  • The contact gate generally stays low during approach, rises during sustained contact, and falls after interaction subsides.Rapid tactile-view appearance changes coincide with gate increases; an episode starting near contact activates the gate earlier, while the gate modulates the coarse bias toward tactile keys rather than attention allocation among tactile tokens.Source: §4.4.2 · pp. 8–9, Figure 7 · p. 8

5 Conclusion · A Additional Experimental Details

Dream-Tac jointly models actions, future visual observations, and tactile dynamics for contact-rich manipulation, using contact-sensitive fusion and attention plus dual-level acceleration. On six real-world tasks, it outperforms strong baselines in success rates and future visual prediction accuracy.

5 Conclusion

  • Dream-Tac jointly models actions, future visual observations, and tactile dynamics to capture physical interactions critical for manipulation.Source: §5 · p. 9
  • Unlike prior world action models that rely mainly on vision, Dream-Tac incorporates tactile dynamics into its unified model.Source: §5 · p. 9
  • Dream-Tac consistently outperforms strong baselines on six real-world contact-rich tasks, with higher success rates and more accurate future visual predictions.Source: §5 · p. 9
  • Contact-gated visuotactile fusion and contact-aware attention bias selectively emphasize tactile information during important contact transitions.Source: §5 · p. 9
  • A dual-level acceleration design makes Dream-Tac more practical for real-time deployment.Source: §5 · p. 9
  • The results demonstrate the effectiveness of unified visuo-tactile world modeling and point to a promising direction for robotic systems in contact-rich environments.Source: §5 · p. 9

A.1 Real-World Experimental Setup · A.2 Task Details

The real-world experiments use a Franka Panda platform with global, wrist-level, and fingertip tactile sensing across six manipulation tasks. The tasks test tactile-guided contact control, with Mahjong relying only on tactile feedback to infer the hand configuration.

A.1 Real-World Experimental Setup

  • The platform combines a Franka Panda, fixed and wrist-mounted RealSense D435i cameras, and two fingertip Xense Photon tactile sensors.The fixed camera captures the global scene, the wrist camera provides local egocentric views, and the tactile sensors capture contact and deformation.Source: Figure 8 · p. 11, §A.1 · p. 11

A.2 Task Details

  • During cucumber peeling, the robot maintains moderate force and continuous blade contact while moving lengthwise, using tactile feedback to avoid losing contact or cutting too deeply.Success is defined as cleanly removing the skin.Source: §A.2 · p. 11
  • For banana cutting, the robot regulates force and maintains a stable cutting direction, using tactile signals about contact onset and resistance to guide the cut.Success requires cutting into the banana and creating a clear separation.Source: §A.2 · p. 11
  • For USB insertion, the robot aligns the device with the port and adjusts its motion using tactile cues about contact onset, alignment errors, and resistance.Success requires fully inserting the USB into the port.Source: §A.2 · p. 11
  • For Mahjong, the robot uses tactile feedback alone to infer the hand configuration, then pushes tiles to indicate a win or places the tile at the table center.Success requires a correct judgment and accurate execution of the corresponding action.Source: §A.2 · p. 11
  • For baguette pick-and-place, the robot regulates its grasp using tactile cues about deformation and interaction dynamics while lifting and transporting the soft object.Success requires placing the baguette fully inside the basket without noticeably moving it.Source: §A.2 · p. 11
  • For whiteboard cleaning, the robot maintains contact and regulates wiping force using tactile cues about contact consistency and friction, then returns the eraser to its designated location.Success requires that the writing is no longer visible and the eraser is returned to its original position.Source: §A.2 · p. 11

A.3 Experimental Settings · A.4 Baseline Settings

The paper describes how demonstrations are collected and evaluated, then specifies four baselines and their training setups. These passages establish experimental protocol and comparison settings, not comparative results.

A.3 Experimental Settings

  • Each task has 100 demonstrations collected with a 3D SpaceMouse, randomized initial states and interactions, and synchronized visual, tactile, and proprioceptive recordings at 30 Hz.Demonstration length varies with the time required to complete the task.Source: §A.3 · p. 11
  • Evaluation runs 20 trials per task with randomized object positions; a trial fails if the task is not completed within the maximum allowed steps.The maximum steps are the maximum execution steps allowed during evaluation.Source: §A.3 · p. 12

A.4 Baseline Settings

  • The comparison includes π0 and π0.5 general-robot-control VLA flow models, ForceVLA for contact-rich manipulation, and Cosmos Policy, a world action model for complex task execution.π0.5 is described as an enhanced VLA flow model for physical reasoning and long-horizon manipulation.Source: §A.4 · p. 12
  • π0 and π0.5 post-training uses one NVIDIA H100 GPU, both camera views, action chunks of 50, and 30,000 steps with batch size 32 and learning rate 5 × 10^-5.Source: §A.4 · p. 13
  • ForceVLA is trained separately for each task on eight NVIDIA H100 GPUs for 30,000 steps, using Adam, a decayed learning rate, bfloat16 precision, and gradient clipping at 1.0.The learning rate peaks at 2.5 × 10^-5 and decays to 2.5 × 10^-6; Adam uses (β1, β2) = (0.9, 0.95).Source: §A.4 · p. 13
  • Cosmos Policy uses Dream-Tac’s training configuration across tasks, including the optimizer, learning-rate schedule, mixed-precision strategy, and checkpoint-selection protocol.The Cosmos Policy models are trained on eight NVIDIA H100 GPUs.Source: §A.4 · p. 13

A.5 Training Hyperparameters · A.6 Contact Gate Statistics · B Implementation Details of the Acceleration Modules

Dream-Tac is fine-tuned with a staged learning-rate schedule and joint action, visual, and tactile denoising. Its deterministic tactile gate suppresses weak changes and increases tactile influence around contact-relevant transitions.

A.5 Training Hyperparameters

  • Dream-Tac is fine-tuned from Cosmos-Predict2-2B Video2World with Fused Adam at learning rate 10−4 and a multiplier warmed up for 2,000 steps, decayed to 0.3 by step 20,000, then fixed at 0.06.Training uses (β1, β2) = (0.9, 0.99), ε = 10−8, weight decay 0.1, and mixed bfloat16 precision.Source: §A.5 · p. 13
  • Action and future-prediction tokens are jointly noised and denoised in one pass under the inherited rectified-flow/hybrid-EDM objective, while observed context latents remain clean.Noise σ uses a hybrid mixture: log-normal (pμ, pσ) = (ln 4, 1.2) on [0.01, 200] and uniform on [1, 85]; text dropout is 0 and EMA is disabled.Source: §A.5 · p. 13
  • Training uses synchronized third-person and wrist RGB at 224×224, normalized proprioception and actions, and 20-step chunks; tactile models encode fingertip RGB with the Wan VAE and predict future tactile latents jointly with visual latents and actions.Per-GPU batch size is 25 for vision-only and 16 for visuo-tactile models.Source: §A.5 · p. 13
  • Contact-aware self-attention uses tactile logit-bias strength α = 2.0, and the final model is selected with the same validation-based rule across methods.Checkpoints are saved periodically; gate hyperparameters and clipping follow Sec. 4.4.2 and Eq. (9).Source: §A.5 · p. 13

A.6 Contact Gate Statistics

  • The tactile gate is computed without a learned gating network: it takes the maximum normalized mean absolute frame difference across left and right fingertip RGB views, with ρ0 = 0, then maps ρt through a fixed sigmoid.The sigmoid uses (m, s, k, ε) = (0.002, 0.001, 4, 10−6) and output range [0.15, 1.0].Source: §A.6 · p. 13
  • Most frame-to-frame tactile changes are small, while rarer larger changes signal salient interactions; the gate stays low for small changes and rises rapidly beyond the typical noise range.These statistics support CASA’s use of a simple deterministic gate to suppress weak tactile perturbations and amplify tactile influence during contact transitions.Source: §A.6 · p. 13, Figure 10 · p. 13

B.1 FlashBias Implementation · B.2 Diffusion-Step Time Cache

Dream-Tac’s FlashBias reformulation incorporates the contact-aware attention bias without breaking fused attention, while its diffusion-step cache exploits redundancy between adjacent steps to reduce repeated inference computation.

B.1 FlashBias Implementation

  • The tactile attention bias is rank-one, Δ_ij = u_i v_j, with factors derived from the gate and binary query/key positions.The factors are u_i = √γ_b a_i and v_j = √γ_b b_j.Source: §B.1 · p. 14
  • Augmenting queries and keys with one scalar channel incorporates the bias exactly into their inner product, avoiding a dense bias matrix.The resulting inner product is equivalent to adding the structured bias to the original attention logits.Source: §B.1 · p. 14
  • FlashBias preserves fused scaled dot-product attention, avoids O(S^2) bias materialization, and adds only O(1) per-token state per head.Zero-padding augmented channels when required for kernel alignment does not affect the logits; the implementation improves training efficiency.Source: §B.1 · p. 14

B.2 Diffusion-Step Time Cache

  • The cache reuses an expensive intermediate computation across similar denoising steps, reducing repeated work with limited approximation error.Rather than recomputing that component at every step, the method caches its output at a selected step for reuse.Source: §B.2 · p. 14
  • Timestep-embedding similarity does not reliably indicate action-latent change, whereas adjacent denoising steps show strong output redundancy.The passage reports that neither relative L1 distance nor cosine similarity between timestep embeddings and latent features reliably indicates action change.Source: §B.2 · p. 14
  • Adjacent diffusion-step outputs have average cosine similarity of approximately 0.997, motivating cache-based inference acceleration.The passage attributes this high similarity to outputs changing only marginally across most denoising steps.Source: §B.2 · p. 14
  • Full forward computation is performed only at the first and third steps, the latter having the largest validation-set variation; cached results are reused at other steps.This design preserves generation quality while substantially reducing inference cost.Source: §B.2 · p. 14

C Additional Reconstruction Results

Additional qualitative reconstructions show Dream-Tac’s predictive capability across visual and joint visuo-tactile modalities, with temporally coherent and physically meaningful future representations.

C Additional Reconstruction Results

C.1 Visual Reconstruction · C.2 Visuo-Tactile Reconstruction

Dream-Tac reconstructs future visual and tactile observations, capturing task-relevant scene dynamics and phase-dependent contact patterns. These reconstructions provide visually coherent and physically grounded predictions for contact-rich manipulation.

C.1 Visual Reconstruction

  • Dream-Tac’s predicted Cut Banana frames preserve scene layout and gripper–knife–banana relationships while capturing coarse future motion.The results indicate a stable action-conditioned representation of future visual dynamics.Source: §C.1 · p. 14
  • Despite smoothed fine textures and high-frequency details, predicted frames remain semantically consistent with ground truth and reflect manipulation progress.The passages characterize these predictions as learned visual structures aligned with task evolution.Source: §C.1 · p. 14
  • Future visual reconstruction serves as an informative intermediate representation for action generation in world action models.Source: §C.1 · p. 15

C.2 Visuo-Tactile Reconstruction

  • Dream-Tac jointly reconstructs future visual and tactile observations in a shared latent space, grounding predictions in touch when RGB weakly reveals contact-state transitions.Joint reconstruction encourages future representations that are both visually coherent and physically grounded.Source: §C.2 · p. 15
  • In Pre-contact, predicted tactile images preserve the undeformed background without spurious contact signals; during Cutting, they reproduce overall deformation and intensity redistribution.Fine pin-level details are smoothed, but the main spatial contact pattern and its phase-dependent change are captured.Source: §C.2 · p. 15
  • Tactile reconstructions distinguish quiescent from contact-active states and predict feedback evolution with manipulation, supporting the importance of touch for precise contact perception.The passage links these results to Dream-Tac’s stronger performance on tasks requiring precise contact perception and fine-grained physical interaction.Source: §C.2 · p. 15

D Limitations and Future Work

Dream-Tac’s evaluation scope, tactile gating, and computational cost leave limitations, motivating broader data, richer contact modeling, greater efficiency, and more challenging manipulation settings.

Limitations

  • Dream-Tac is tested on limited real-world tasks, uses simple frame-to-frame tactile variation for gating, and remains computationally expensive relative to lightweight reactive policies.Generalization across broader task families, more diverse objects, and complex environments remains unverified; subtle or long-horizon interaction patterns may not be captured by the current gating mechanism.Source: D Limitations and Future Work · p. 15

Future Work

  • Future work will scale Dream-Tac to a larger, more diverse visuo-tactile dataset, explore more expressive contact modeling, and improve efficiency for faster real-time control.Source: D Limitations and Future Work · p. 15
  • Future work will extend Dream-Tac to dexterous multi-stage manipulation, deformable object interaction, and long-horizon contact-rich tasks.Source: D Limitations and Future Work · p. 15

Related papers

Loading 2606.08737v1…