PaperlayerView exact revision on arXiv

Source-linked AI summary

VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation

Shuai Tian, Yupeng Zheng, Yuhang Zheng, Songen Gu, Yujie Zang, Yuxing Qin, Weize Li, Haoran Li, Wenchao Ding, Dongbin Zhao

arXiv:2607.02503v1cs.RO

TL;DR

Contact-rich manipulation requires transient tactile information that is often weakly visible in vision, while existing policies rarely model tactile deformation dynamics during action generation. VT-WAM jointly predicts future visual states, tactile deformation, and actions with flow matching, using asymmetric attention and contact-gated tactile guidance. Across six real-world tasks, it achieves 71.67% average success and outperforms Fast-WAM by 26.67%.

  • Problem

    Existing visual-tactile policies often feed tactile observations into action prediction without modeling tactile deformation dynamics, despite contact-rich manipulation relying on sparse local interaction cues.

  • Method

    VT-WAM jointly learns future visual prediction, tactile deformation prediction, and action prediction with unified flow matching, using Asymmetric MoT Attention and contact-gated AVTAG.

  • Results

    71.67% average success was achieved across six real-world tasks, outperforming Fast-WAM by 26.67%.

  • Takeaways & Limitations

    Modeling tactile deformation as interaction dynamics and guiding contact-phase tactile attention support action prediction in contact-rich tasks.

Abstract

from arXiv · show

Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.

I. INTRODUCTION

VT-WAM addresses the underuse of sparse tactile signals in contact-rich manipulation by coupling tactile deformation dynamics with action prediction. It combines joint visual-tactile-action flow matching with asymmetric attention and contact-gated tactile guidance, achieving strong results across six real-world tasks.

  • Motivation: Contact-rich manipulation depends on transient deformation, pressure, slip, and friction cues that are weakly visible or occluded in vision.These local interaction states make vision-centric policies unreliable when execution requires tactile-informed adjustments.
  • Motivation: Tactile responses occur mainly during brief contact phases, whereas visual observations provide dense information across most frames.This temporal imbalance can cause jointly trained networks to favor visual evidence and underuse tactile signals.
  • Approach: VT-WAM jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework.The formulation couples tactile deformation dynamics with action prediction rather than treating tactile observations only as policy inputs.
  • Approach: Asymmetric MoT Attention connects a first-frame visual anchor and full tactile sequence to action queries, while contact-gated AVTAG encourages tactile attention during contact phases.AVTAG uses training-only hinge ranking guidance without changing the inference-time architecture.
  • Results: 71.67% average success was achieved across six real-world tasks, outperforming Fast-WAM by 26.67%.The evaluation covers surface-interaction and constrained insertion regimes, and ablations support both core designs.

II. RELATED WORK

Related work spans tactile-conditioned and reactive policies, visual World Action Models, and multimodal architectures. VT-WAM extends joint World Action Models by adding tactile deformation dynamics and connects visual, tactile, and action experts through asymmetric attention.

  • Tactile policies: Tactile policies commonly condition diffusion or reactive control systems on tactile or force feedback for contact correction.Examples include FARM, TacDiffusion, PolyTouch, KineDex, RDP, Force Policy, and M2-ResPoli.
  • World Action Models: World Action Models incorporate future-state prediction into action prediction and include cascaded and joint architectures.Joint WAMs learn future dynamics and actions within a shared generative objective, as in Fast-WAM and related methods.
  • VT-WAM: VT-WAM extends visual World Action Models by jointly predicting future visual states, tactile deformation, and actions within unified flow matching.Its inputs include wrist-camera observations, tactile deformation observations, proprioceptive state, language, and action chunks.
  • VT-WAM: The architecture uses visual, tactile, and action experts, with Asymmetric MoT Attention connecting their token streams for joint prediction.The visual expert provides scene context, the tactile expert models local contact evolution, and the action expert predicts action chunks.
  • VT-WAM: Masked attention determines which visual, tactile, and action query tokens can attend to which key tokens, producing updated multimodal features.Projection heads then predict modality-specific velocity fields under the flow matching objective.

B. Asymmetric MoT Attention

Asymmetric MoT Attention routes information asymmetrically so visual tokens provide scene context, tactile tokens model contact evolution, and action tokens access both during control.

  • B. Asymmetric MoT Attention: During training, visual, tactile, and action branches remain in one joint flow-matching model, while inference removes future visual tokens.
  • B. Asymmetric MoT Attention: The visual expert is protected from tactile and action keys, keeping visual representations focused on scene context.
  • B. Asymmetric MoT Attention: The tactile expert sees only the first-frame visual anchor among visual tokens, grounding tactile dynamics in scene context without future visual dependence.
  • B. Asymmetric MoT Attention: Asymmetric MoT Attention gives action prediction access to the first-frame visual anchor and the full tactile sequence.This matches the visual-cache inference mode used for control.
  • B. Asymmetric MoT Attention: The resulting readout stabilizes visual representations, grounds tactile dynamics visually, and supplies action prediction with contact-evolution information.

C. Contact-Gated Action-Visual-Tactile Attention Guidance

Contact-gated AVTAG addresses visual–tactile information imbalance by training action queries to favor tactile evidence when deformation indicates physical contact.

  • C. Contact-Gated Action-Visual-Tactile Attention Guidance: Visual observations are dense across frames, whereas tactile deformation is local and informative mainly during short contact intervals.
  • C. Contact-Gated Action-Visual-Tactile Attention Guidance: AVTAG adds a training-only auxiliary objective that compares action-query attention to visual and tactile evidence.It does not directly change the visual or tactile key representations.
  • C. Contact-Gated Action-Visual-Tactile Attention Guidance: The auxiliary attention sums action-token attention to visual and tactile keys and normalizes the totals into relative weights.
  • C. Contact-Gated Action-Visual-Tactile Attention Guidance: AVTAG applies its hinge ranking loss only to action tokens in contact phases identified by pronounced tactile deformation.
  • C. Contact-Gated Action-Visual-Tactile Attention Guidance: The loss penalizes visual-dominant attention during contact and stops penalizing once tactile attention is at least visual attention.

1) Flow Matching Training Objective:

VT-WAM trains visual, tactile, and action experts jointly with flow matching, while evaluation spans six contact-rich tasks across two interaction regimes.

  • 1) Flow Matching Training Objective:: VT-WAM uses a joint flow-matching objective over visual, tactile, and action tokens.Each modality expert predicts the velocity field for its corresponding tokens.
  • 1) Flow Matching Training Objective:: The visual, tactile, and action experts predict velocity fields against corresponding flow-matching targets.
  • 1) Flow Matching Training Objective:: AVTAG is incorporated into the full training objective when enabled.
  • 1) Flow Matching Training Objective:: The evaluation covers six real-world tasks spanning surface-interaction and constrained insertion regimes.

2) Efficient Visual-Cache Inference:

VT-WAM uses joint inference for multimodal prediction analysis and visual-cache inference for real-world control, retaining the current visual observation as an anchor while denoising tactile and action latents.

  • 2) Efficient Visual-Cache Inference:: Joint inference denoises visual, tactile, and action tokens together to assess visual-tactile prediction.
  • 2) Efficient Visual-Cache Inference:: Visual-cache inference keeps the current visual observation as a first-frame anchor, removes future visual prediction, and denoises only tactile and action latents.

IV. EXPERIMENTS

The experiments evaluate VT-WAM on real-world contact-rich manipulation using a multimodal robotic platform, standardized demonstrations, and comparisons with visuomotor, VLA, and WAM baselines.

  • Evaluation: The experiments include task performance, visual-tactile prediction quality, and ablation studies measuring contributions from tactile dynamics modeling and attention guidance.The setup description states that these analyses quantify the contribution of key components.
  • Experimental setup: Training uses 100 expert trajectories per task, with synchronized visual, tactile, proprioceptive, and action streams resampled to 30 Hz.The model uses pretrained Wan2.2-5B for vision and 1B-scale DiT models for tactile and action experts.
  • Baselines: The evaluation compares VT-WAM with tactile-conditioned diffusion, reactive visual-tactile, vision-language-action, and visual world action model baselines.Fast-WAM models visual dynamics and predicts actions without tactile input, while OmniVTLA uses tactile observations for action prediction.

B. Benchmark Tasks and Evaluation Metrics

The benchmark covers six contact-rich tasks divided into surface interaction and constrained insertion regimes, with success rates computed from 20 independent trials per method and task.

  • Benchmark tasks: The six tasks comprise wipe board, wipe vase, and peel cucumber for surface interaction, plus insert plug, swipe card, and insert tube for constrained insertion.Surface tasks require sustained motion across planar, curved, or deformable surfaces, whereas insertion tasks require fine alignment under geometric constraints or occlusion.
  • Evaluation metrics: Each method and task is evaluated over 20 independent trials, and the reported metric is the success rate.Surface-interaction scores are 0, 0.5, or 1, while constrained-insertion scores are binary.
  • Evaluation metrics: Surface-interaction scores reflect failure, completing more than half the target region, or completing the full target region.The corresponding values are 0, 0.5, and 1.
  • Evaluation metrics: Constrained-insertion success is 1 when the object reaches the target position and 0 otherwise.This regime uses a binary success metric.

C. Main Results

VT-WAM achieves the strongest reported performance across contact-rich tasks and predicts tactile deformation dynamics that capture local contact patterns. Ablations and prediction-quality analysis support the contribution of tactile dynamics modeling and attention design.

  • Task performance: 71.67% average success is achieved by VT-WAM, improving on Fast-WAM’s 45.00% by 26.67% across six contact-rich tasks.Table I reports VT-WAM as the highest-performing method among all evaluated methods.
  • Task performance: 81.67% success is achieved by VT-WAM on surface-interaction tasks, compared with 56.67% for Fast-WAM, 36.67% for π0.5, and 33.33% for OmniVTLA.The passage attributes the result to modeling tactile deformation as interaction dynamics.
  • Task performance: 61.67% success is achieved by VT-WAM on constrained insertion tasks, compared with 38.33% for OmniVTLA and 33.33% for Fast-WAM.These tasks require fine alignment, and the insert-tube setting makes visual alignment unreliable because the tube is transparent.
  • Visual-tactile prediction: VT-WAM predicts temporally coherent wrist-camera observations and tactile deformation trajectories that capture pressure concentration and contact migration.The analysis uses joint inference for visual-tactile prediction, while real-world control uses visual-cache inference.
  • Ablation study: The ablation variants separately test symmetric tactile-sequence attention, first-frame tactile attention, and full tactile-sequence attention without AVTAG.VT-WAM is the full model in this ablation comparison.
  • Visual-tactile prediction: VT-WAM achieves lower deformation error and higher directional consistency than baseline models in tactile deformation prediction.The reported metrics indicate that the tactile expert learns meaningful contact deformation dynamics.

D. Ablation Studies

The ablations show that temporal tactile dynamics improve action prediction, while contact-gated AVTAG further improves success by encouraging tactile attention during contact-sensitive execution. Across the studied tasks, the conclusion supports modeling tactile deformation as interaction dynamics for contact-rich manipulation.

  • Ablation 1: 70% and 50% success with full tactile sequences exceed 40% and 30% using only the first tactile frame on wipe vase and insert tube, respectively.The comparison isolates the benefit of temporal tactile information for action prediction.
  • Ablation 1: Adding symmetric tactile-sequence prediction raises success from 55% to 65% on wipe vase and from 25% to 40% on insert tube versus visual-only Fast-WAM.The result shows that tactile dynamics provide information beyond visual dynamics, although symmetric fusion increases inference cost.
  • Ablation 2: Contact-gated AVTAG increases success from 70% to 85% on wipe vase and from 50% to 55% on insert tube over the same Asymmetric MoT model without AVTAG.Both variants use the same Asymmetric MoT Attention, so the comparison isolates training-time AVTAG guidance.
  • Ablation 2: AVTAG addresses contact recovery when the wrist camera changes only subtly after contact is lost, encouraging greater reliance on tactile evidence.The side view is used only to visualize the interaction, while the wrist camera is the policy’s only visual input.
  • Overall conclusion: The conclusion reports consistent improvements over visual-only and tactile-input baselines and identifies tactile dynamics modeling and contact-phase tactile use as important.The experiments cover surface-interaction and constrained-insertion tasks, while multi-task training remains unexplored.
Loading 2607.02503v1…