Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention

1 Introduction

• Slip, jamming, contact-direction changes, and subtle misalignment can be hard to observe in RGB yet determine corrective actions, motivating future tactile-state prediction for WAMs.Source: §1 · p. 2
• Tactile pollution occurs when unconstrained video attention to sparse tactile signals interferes with pretrained visual representations, degrading video and action prediction.Naive tactile fusion can blur and distort predicted visual futures.Source: §1 · p. 2
• Tactile pixel changes correlate only weakly with deformation-derived contact similarity, so effective modeling must identify important touch and preserve contact-relevant dynamics.Small pixel changes can accompany slip, compression, or slight in-gripper rotation, while large changes can reflect physically minor variation.Source: §1 · p. 2
• TAAM blocks video queries from tactile keys while preserving tactile access for action queries, using observed proxy changes for attention bias and future-proxy supervision for contact dynamics.The six-dimensional touch-aware proxy is derived from optical flow between consecutive tactile images.Source: §1 · p. 2
• Tactile-WAM is validated on nine simulation tasks and five real-robot tasks, with substantial contact-rich manipulation improvements while preserving visual prediction quality.Source: §1 · p. 2

World Action Models

• WAMs jointly predict future visual observations and action chunks, conditioning action generation on environment evolution (Ye et al. 2026; Guo et al. 2024; Hu et al. 2024).At policy-call time, they receive RGB and proprioceptive histories and a language instruction; future RGB is encoded into visual latents.Source: World Action Models · p. 2
• Using conditional flow matching, WAMs transform Gaussian noise into future visual latents and executable actions at inference.The model predicts vector fields for future visual latents and actions, then inference transforms sampled noise into those outputs.Source: World Action Models · p. 2

Vision-Based Tactile Sensing

• Vision-based tactile sensors use an internal camera to capture contact-induced deformation; their images convey local geometry, pressure, and temporal dynamics such as shear and slip, but not direct force values.Unlike calibrated force measurements, tactile images are high-dimensional visual observations.Source: Vision-Based Tactile Sensing · p. 2

Tactile Pollution

• Naive VT-WAM’s unrestricted tactile injection caused Tactile Pollution: on UniVTAC, it had 50% higher future-frame prediction MSE than DreamZero and visible blur around manipulated objects.On 800 training trajectories, it also consistently had higher training loss. Shared multimodal tokens let visual queries attend to tactile keys, and fine-tuning with limited tactile data interfered with the visual dynamics prior.Source: Tactile Pollution · p. 3, Figure 2 · p. 3

Pixel–Contact Misalignment

• Because tactile sensing captures local contact states and transitions, pixel-level changes are an unreliable proxy, and pixel-reconstruction objectives alone cannot ensure action-relevant contact changes are preserved.The relevant transitions include slip, compression, and object motion.Source: Pixel–Contact Misalignment · p. 3
• Across 5,000 tactile image pairs, pixel variation showed no clear linear correlation with contact variation, with examples where either changed substantially while the other did not.Some pairs had similar pixel variation but substantially different contact changes, and vice versa.Source: Pixel–Contact Misalignment · p. 3, Figure 3 · p. 3

Architecture Overview

• The language-conditioned Wan DiT jointly predicts future video frames, tactile states, and action chunks from RGB and tactile histories encoded by a frozen Wan2.2 VAE.Modality embeddings are added to the separate visual and tactile latents.Source: Architecture Overview · p. 3

Joint Visual-Tactile-Action Modeling

• Historical tactile latents are projected and added as auxiliary conditioning; video, action, and tactile flow-matching losses are jointly optimized, with actions aligned to tactile latent frames.Predicted tactile latents are decoded into tactile images.Source: Joint Visual-Tactile-Action Modeling · p. 4

Tactile Asymmetric Attention Mask

• The asymmetric mask blocks visual queries from tactile keys to prevent tactile pollution, while action queries retain tactile access and tactile queries preserve multimodal context.A touch-aware bias strengthens action-query attention to tactile representations within causal blocks, computed from observed tactile forces rather than future targets or denoising predictions.Source: Tactile Asymmetric Attention Mask · pp. 4–5

Touch-aware Proxy

• A differentiable deformation proxy converts predicted tactile-image transitions into proxy signals, which are supervised against signals derived from ground-truth tactile trajectories.Each action chunk corresponds to eight tactile-transition frames; the proxy summarizes tangential deformation and average divergence.Source: Touch-aware Proxy · p. 5

Experimental Setup

• Experiments compare π0.5, DreamZero, and Naive VT-WAM across eight UniVTAC tasks, nine ManiFeel tasks, and six real-robot conditions under fixed trial budgets.π0.5 uses released pretrained weights, while WAM variants are trained from scratch.Source: Experimental Setup · p. 5

Simulation Results

• Tactile-WAM achieves 147/450 (32.7%) on ManiFeel, improving over DreamZero by 17.1 percentage points, while π0.5 reaches 36.7% overall.It performs best on peg insertion, gear insertion, bulb insertion, and bolt-nut assembly.Source: Simulation Results · p. 5

Real-Robot Results

• Tactile-WAM succeeds in 59/120 real-robot trials (49.2%), exceeding π0.5 (24.2%), DreamZero (21.7%), and VT-WAM (15.8%).It ranks first in all six conditions, retaining 50% success on dim-light power insertion; DreamZero drops from 55% to 20% there.Source: Real-Robot Results · pp. 5–6, Figure 5 · p. 6

Tactile Pollution and VideoClean

• VideoClean reduces prediction-to-DreamZero MSE and MAE by 21.8% and 17.8%, and is closer on 84.5% and 90.5% of samples, respectively.At H2–H4, MSE reductions are 3.3%, 22.4%, and 38.5%; all confidence intervals for ground-truth prediction quality include zero.Source: Tactile Pollution and VideoClean · p. 6, Table 3 · p. 7, Table 4 · p. 7

Effect of Contact-Aware Attention Bias

• During bulb insertion, contact-aware bias redirects action attention to tactile keys after initial contact, enabling correction before RGB shows the error; without bias, residual contact develops into slip.The comparison supports the mechanism but does not isolate the bias’s success-rate contribution because no bias-only real-robot variant is tested.Source: Effect of Contact-Aware Attention Bias · pp. 6–7, Figure 7 · p. 7

Ablation Study

• VideoClean raises success from 15.1% to 28.4% (+13.3 percentage points), and adding virtual-force supervision with contact-aware bias raises it to 32.7% (+4.3 points).Naive VT-WAM reaches 15.1%, slightly below DreamZero’s 15.6%; full Tactile-WAM improves over DreamZero from 15.6% to 32.7% (+17.1 percentage points; 2.10×). Force supervision and bias are introduced together, so their individual effects are not identified.Source: Ablation Study · p. 7, Table 6 · p. 7

• 32.7% success and a 17.1-point improvement over DreamZero accompany state-of-the-art results on UniVTAC and ManiFeel, alongside a 21.8% reduction in visual trajectory MSE.Source: §6 · p. 7
• Physical evaluation ranks Tactile-WAM first across all conditions.Source: §6 · p. 7
• Tactile-WAM addresses tactile pollution through asymmetric attention and a deformation-based proxy.Source: §6 · p. 7

A Related Work

• Tactile-WAM treats touch as a predicted physical state and constrains its interaction with video and action tokens, distinguishing it from prior world-model and tactile-learning approaches.Related directions include joint observation-action denoising (Guo et al. 2024; Hu et al. 2024; Ye et al. 2026), transferable tactile representations (Xu et al. 2024; Higuera et al. 2024), separate transformer pathways (Li et al. 2026), and training-only auxiliary heads (Zhang et al. 2026); Tactile-WAM also supervises decoded tactile latents.Source: A Related Work · p. 10

B Model Details

• The model architecture comprises a shared latent representation, multimodal tokenization, causal attention, touch-aware routing, prediction heads, and an optimization configuration.This overview frames the model details described in the subsequent subsections.Source: B Model Details · p. 10

Backbone and Latent Representation

• RGB and optical tactile observations share a frozen Wan2.2 VAE that maps both into a 48-channel latent space, with the two tactile sensors arranged as a horizontal mosaic.The shared VAE provides a common generative latent space without a separate tactile tokenizer.Source: Backbone and Latent Representation · p. 10

Multimodal Tokenization

• Video tokens use native three-dimensional rotary positional encoding (3-D RoPE), while tactile, action, and state tokens use one-dimensional RoPE; causal block ordering and masking preserve temporal correspondence.Source: Multimodal Tokenization · p. 10

Temporal Block Organization

• The model encodes 33 image frames into nine latent frames, retaining the first as a clean condition and splitting the remaining eight into four two-frame causal blocks.Each block aligns with a 24-step action chunk and one proprioceptive state token.Source: Temporal Block Organization · pp. 10–11

Causal Multimodal Attention

• VideoClean blocks visual queries from reading tactile keys while preserving action queries’ access to tactile keys and causally permitted context.The touch-aware bias strengthens action-to-tactile attention only within already accessible causal blocks, so it cannot expose future tokens.Source: Causal Multimodal Attention · p. 11

Touch-Aware Proxy

• Future tactile proxy supervision uses decoded tactile latents, while the attention bias uses a single nonadjacent observed-proxy difference rather than a sum of eight differences.The proxy operator has no learnable parameters, remains differentiable with respect to predicted latents, and uses a frozen VAE.Source: Touch-Aware Proxy · p. 11

Prediction Heads and Objective

• The objective jointly predicts visual latents, tactile latents, and actions, with modality-specific losses normalized over their own feature dimensions.The tactile output head maps each token from 3072 to 192 dimensions before unpatchifying it into a 48-channel latent flow field; the action head is a two-layer MLP.Source: Prediction Heads and Objective · p. 11

Optimization

• The DiT backbone, multimodal projections, modality embeddings, modified attention layers, and prediction heads are jointly fine-tuned with AdamW, while the pretrained VAE and language encoder remain frozen.Optimization uses a peak learning rate of 1 × 10−5, linear warmup over the first 5% of optimization, and cosine decay.Source: Training Configuration · p. 11

Flow-matching construction

• Training uses 1000 discrete noise levels, with one timestep sampled uniformly per causal block and shared across temporally aligned visual, tactile, and action targets.The first conditioning frame remains clean, and the scheduler weights are normalized to unit mean.Source: Training Configuration · p. 12

Loss balancing and numerical stability

• Fixed modality coefficients weight timestep-weighted mean-squared-error losses, with action padding and unavailable targets masked before reduction and each branch averaged over valid tokens and feature dimensions.Training uses BF16 mixed precision, TF32 matrix multiplications, activation recomputation, global gradient-norm clipping at 1.0, and random seed 42.Source: Training Configuration · p. 12

Training scale and hardware

• Training runs on eight NVIDIA A100 GPUs with micro-batch size 1 per device and one gradient-accumulation step, yielding an effective global batch size of 8.DeepSpeed ZeRO-2 shards optimizer states and gradients across devices without CPU optimizer offloading.Source: Training Configuration · p. 12

Experimental Setup

• ManiFeel evaluation uses 60K training steps and 450 trials per method across nine tasks; UniVTAC uses 100K steps and 160 trials across eight tasks.The ManiFeel ablation study uses the same 60K-step budget.Source: Experimental Setup · p. 12
• For ManiFeel, observations are sampled at 10 fps, and the policy predicts 24 actions and executes 8 before replanning.The simulator runs at 60 Hz with control decimation 4; six-dimensional task actions are zero-padded to the model’s 7-D interface and unpadded before execution.Source: Experimental Setup · p. 12
• The real-robot evaluation covers five tasks, with power insertion also tested under dim lighting, for six conditions and 120 trials per method.Each condition has 20 trials; hardware and procedural details are provided in the supplementary material.Source: Experimental Setup · p. 12
• Comparisons include π0.5, DreamZero, and Naive VT-WAM; only from-scratch WAM variants provide the most controlled measure of tactile modeling.π0.5 is fine-tuned from released base weights, while DreamZero, Naive VT-WAM, and the proposed variants are trained from scratch; success rate is the primary metric, reported as successful executions over fixed trial counts.Source: Experimental Setup · p. 12

Qualitative Execution Trajectories

• Successful execution visualizations preserve temporal order, task-specific stages, synchronized camera views, and tactile observations.Source: Qualitative Execution Trajectories · p. 12

Qualitative Results on Simulation Tasks

• Figures S1–S17 show successful simulation trajectories: S1–S9 cover ManiFeel’s nine tasks, and S10–S17 cover UniVTAC’s eight tasks.ManiFeel views include front, side, wrist, and bilateral visuo-tactile observations; UniVTAC views include external, wrist, and bilateral visuo-tactile observations.Source: Qualitative Results on Simulation Tasks · p. 12

Qualitative Results on Real-Robot Experiments

• The sequences show successful bulb seating and tightening, gear meshing with functional-contact verification, and nut threading and tightening.The bulb is rotated while grasped; the gear is rotated after engagement; the nut sequence includes seating and thread alignment.Source: Figure S18 · p. 26, Figure S19 · p. 26, Figure S20 · p. 27
• Other successful sequences show peg insertion and phone pick-and-place through grasp, transport, placement, and release.The peg is aligned with the socket before insertion; the phone is lifted and transported before placement.Source: Figure S21 · p. 27, Figure S22 · p. 28
• Power-plug insertion succeeds under both dim and standard lighting, with contact-guided socket search and alignment shown under dim lighting.Both sequences include insertion and final verification.Source: Figure S23 · p. 28, Figure S24 · p. 29