Source-linked AI summary
VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs
Haoran Yuan, Weigang Yi, Zhenyu Zhang, Wendi Chen, Yuchen Mo, Jiashi Yin, Xinzhuo Li, Xiangyu Zeng, Chuan Wen, Cewu Lu, Katherine Driggs-Campbell, Ismini Lourentzou
TL;DR
Contact-rich manipulation remains difficult because visual representations provide limited access to fine-grained contact dynamics, while existing tactile integration can allow visual features to dominate. VTAM introduces predictive joint visuo–tactile modeling with action-level virtual-force regularization, achieving strong success across real-world manipulation tasks, including a 90% chip pick-and-place rate.
Problem
Visual representations provide limited access to contact dynamics, and tactile integration can suffer modality collapse when action losses are minimized using visual cues alone.
Method
VTAM jointly predicts future visual and tactile streams in a pretrained video backbone and uses deformation-derived virtual-force prediction to preserve tactile supervision during action training.
Results
VTAM achieves a 90% success rate on chip pick-and-place and outperforms vision-only, regularization-removed, and naive force-integration baselines on contact-rich tasks.
Takeaways & Limitations
Predictive visuo–tactile representation learning combined with action-level regularization improves stability and task success in contact-rich manipulation.
Abstract
from arXiv · showhide
Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions. Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone. In particular, fine-grained force modulation and contact transitions are not reliably encoded in visual tokens, leading to unstable or imprecise behaviors. To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal. VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile-language paired data or independent tactile pretraining. To stabilize multimodal fusion, we introduce a tactile regularization loss that enforces balanced cross-modal attention, preventing visual latent dominance in the action model. VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90 percent on average. In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi 0.5 baseline by 80 percent. Our findings demonstrate that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models.
1 INTRODUCTION
VTAM addresses the difficulty of modeling contact-rich manipulation by treating tactile sensing as a primary modality within a predictive visuo–tactile world model. It combines joint future-stream prediction with action-level regularization to improve tactile sensitivity and task success.
- Motivation: Vision-centric and tactile-augmented VLA designs struggle to represent contact physics because tactile inputs are often mapped into semantic spaces or fused only downstream.These approaches burden the model with inferring contact dynamics within representations optimized for visual alignment or static scene descriptions.
- Approach: Jointly predicting future visual and tactile streams enables temporally consistent contact features without explicit semantic annotations of contact events.The formulation conditions prediction on the robot’s end-effector state.
- Approach: VTAM uses action-level regularization to prevent visual features from dominating optimization and suppressing tactile information.The design maintains sensitivity to tactile signals during action optimization.
- Results: 90% success rate was achieved on chip pick-and-place, compared with 0% for the vision-only baseline and 10% without virtual-force regularization.A naive downstream force integration without predictive visuo–tactile modeling also achieved 0% success.
- Approach: VTAM integrates high-resolution tactile sensing with visual observations in a predictive video backbone for contact-rich robotic manipulation.The model treats touch as a primary sensory modality rather than mapping it into a language-aligned semantic space.
2 RELATED WORKS
Prior work spans vision-language-action models, generative world models, tactile representation learning, and policy-level tactile fusion. VTAM differs by modeling visual–tactile dynamics jointly and preserving tactile influence through deformation-aware virtual-force regularization.
- Vision-Language-Action Models: VLA research extends generalist robot control with geometric priors, hierarchical planning, and predictive world knowledge.These developments aim to improve generalization and sample efficiency.
- Generative World Models for Robotics: Generative robotic world models forecast future environment states or jointly diffuse video and action trajectories for planning and policy learning.The cited approaches build predictive world knowledge on video-based backbones or modality-specific diffusion processes.
- Generative World Models for Robotics: Visual forecasting provides only indirect access to slip, deformation, and force transfer at contact interfaces, which may be weakly observable or hidden from cameras.This limits visual-only world models in contact-rich manipulation.
- Tactile Integration in Robotic Learning: Tactile learning methods align visual and tactile embeddings or incorporate touch through force-aware routing and feedback fusion.These approaches operate at representation or policy levels rather than necessarily modeling joint temporal dynamics.
- VTAM Overview: Alternating intra-view and cross-view attention jointly models multi-view visual and tactile latents before action, virtual-force, and proprioceptive-state prediction.The resulting multimodal representation is injected into a conditional action diffusion head.
- Tactile Integration in Robotic Learning: VTAM embeds tactile perception into a generative video backbone for joint dynamics prediction and adds deformation-aware virtual-force regularization at the action head.The regularization maintains tactile gradient influence without external force-torque sensors.
3. Method
VTAM builds a visuo-tactile world model that jointly predicts visual and tactile dynamics, then uses deformation-derived virtual-force supervision to preserve tactile influence during action learning. Its two-stage training first aligns multimodal latent dynamics and subsequently performs joint action–state–force denoising.
- 3.1. Vision–Tactile Latent World Modeling via Multi-View Diffusion: VTAM embeds visual observations and high-resolution tactile streams into a shared latent space to predict their future evolution.The model uses a pretrained video backbone and treats touch as a primary sensory modality rather than mapping it into a language-aligned semantic space.
- 3.1. Vision–Tactile Latent World Modeling via Multi-View Diffusion: Fine-grained tactile deformation details encode shear, slip, and pressure, which standard semantic vision encoders often discard.VTAM therefore uses a reconstruction-oriented video VAE and embeds tactile streams directly into the video transformer.
- 3.1. Vision–Tactile Latent World Modeling via Multi-View Diffusion: Alternating intra-view and cross-view attention captures modality-specific spatial structure and inter-modal interactions across the visual and tactile streams.Self-attention is first applied independently within each view, followed by attention over concatenated tokens from all views.
- 3.2. Deformation-Aware Regularization via Virtual Force Prediction: Modality collapse can cause action training to ignore tactile feedback when visual cues alone minimize task loss, destabilizing force-sensitive control.VTAM addresses this by supervising a deformation-aware virtual-force signal derived from tactile optical flow rather than requiring external force-torque sensors.
- 3.2. Deformation-Aware Regularization via Virtual Force Prediction: The virtual force proxy is jointly predicted with future actions and states, binding control gradients to tactile representations and maintaining balanced multimodal gradients.The proxy uses tangential flow expectations for shear and flow divergence for normal compression, serving as a geometrically grounded rather than calibrated physical force.
- 3.3. Optimization Objective: VTAM separates training into multimodal latent-flow matching and regularized action prediction to avoid distributional shifts that degrade tactile latents and destabilize convergence.Stage II concatenates action, virtual force, and proprioceptive state in a conditional flow-matching target, with action a ∈ R7, force f ∈ R3, and state s ∈ R16.
4. Experiments
Experiments evaluate VTAM on real-world contact-rich manipulation using synchronized visual and tactile data, comparing it with vision-only and multimodal baselines. VTAM maintains strong performance and more stable contact-aware behavior across chip handling, peeling, and wiping tasks.
- Experimental Setup: VTAM is evaluated on chip pick-and-place, cucumber peeling, and whiteboard wiping tasks requiring fine-grained contact and force control.The dataset contains manually teleoperated trajectories with synchronized multi-view RGB, tactile deformation, and robot-state information.
- Quantitative Results: 90%, 85%, and 95% success rates are achieved by VTAM on chip pick-and-place, cucumber peeling, and whiteboard wiping, respectively.Each task uses 20 trials, with 80 real-world trials per model overall.
- Quantitative Results: On chip pick-and-place, VTAM reaches 90% success while baselines often fail to verify grasps before placement.The task requires force-sensitive handling of fragile chips under visual occlusion.
- Quantitative Results: VTAM reaches 85% success on cucumber peeling, whereas all baselines fail to complete the task.The comparison concerns stable surface contact and shear-force regulation on a deformable object.
- Quantitative Results: VTAM achieves 95% success on flat and tilted whiteboards, while baselines show unstable forces or inconsistent surface following.Qualitative examples attribute baseline failures to vision-driven height and contact errors.
- Prediction Visualization: VTAM predictions preserve cross-view temporal consistency and in-view dynamics, with only minor blurring in manipulation-irrelevant details.Figure 5 compares ground-truth and predicted camera and tactile streams.
5 CONCLUSION
The ablation study tests VTAM’s components on contact-sensitive chip pick-and-place at 1 Hz inference. Full VTAM succeeds substantially more often than variants without predictive visuo-tactile modeling or virtual-force regularization.
- Ablation Study: 90% success is achieved by full VTAM, compared with 0% for vision-only and late-fusion variants and 10% without regularization.The study uses the chip pick-and-place task at a constrained 1 Hz inference frequency.
- Ablation Study: Removing virtual-force regularization reduces success to 10% because vision modality dominance suppresses tactile influence during training.The auxiliary loss is described as preventing representation collapse across the control pathway.
- Ablation Study: Late-stage tactile injection yields 0% success, indicating that raw force input is insufficient without hierarchical visuo-tactile world modeling.The vision-only variant also achieves 0% because occlusion impairs visual depth and contact-transition estimation.
5. Conclusion
VTAM jointly predicts visual and tactile futures so action generation can use contact dynamics that vision may weakly observe or miss. A virtual-force objective maintains tactile supervision and supports stable manipulation on force-sensitive tasks.
- Conclusion: VTAM is a visuo-tactile world action model for contact-rich manipulation.Its predictive backbone models multi-view video and high-resolution tactile signals jointly.
- Conclusion: Joint prediction of future visual and tactile streams learns temporally consistent contact features without explicit contact-event labels.The formulation avoids relying only on downstream reactive tactile fusion.
- Conclusion: A deformation-derived virtual-force objective counteracts visual dominance and keeps tactile supervision active through the control pathway.The objective is introduced to prevent modality collapse during action training.
- Conclusion: VTAM improves success rate and interaction stability over vision-only and naive tactile baselines on force-sensitive robot tasks.The stated examples include chip pick-and-place and cucumber peeling.
A TRAINING DETAILS
Training uses a two-stage video-backbone procedure followed by task-specific optimization, with shared flow-matching objectives and fixed loss weighting. The implementation relies on multi-GPU mixed-precision training and standard optimizer settings.
- Training Procedure: VTAM training uses two stages: video-only pretraining from a GE-base checkpoint followed by subsequent model training.Stage 1 fine-tunes the LTX-Video transformer for 50,000 steps on 4 NVIDIA A100 GPUs.
- Optimization Details: All three objectives receive equal coefficients, λ1 = λ2 = 1, because their mean-squared-error flow-matching losses have comparable scales.Equal weighting avoids introducing additional hyperparameters.
- Baseline Training: The GE-Act baseline follows the same two-stage training protocol as VTAM.Its implementation uses AdamW, warmup, gradient clipping, and a constant learning rate after warmup.
- Baseline Training: π0.5 policies are fine-tuned from a pretrained checkpoint with FSDP, bfloat16 mixed precision, AdamW, cosine decay, and 30,000 optimization steps.Images are resized to 224 × 224 and states and actions use quantile normalization.
B. Experimental Details
VTAM is evaluated in real-world contact-rich manipulation under randomized conditions, with task-specific success criteria across chip handling, wiping, and peeling.
- Policies are evaluated using real-world success rates over independent trials with randomized initial conditions and task-specific criteria.
- Chip pick-and-place uses 20 trials with randomized chip positions, counting failures from missed grasps, breakage, or drops.
- Whiteboard wiping tests flat and 45° inclined boards, allowing at most five motions and requiring complete stain removal without disturbing the board or supports.
- Cucumber peeling requires 20 consecutive motions at a fixed cutting position while dynamically adjusting contact force as cucumber height decreases.
- 85% success (17/20 trials) produced peel strips longer than 10 cm despite changing geometry.
C. Video Prediction Examples
VTAM prediction examples compare generated visual and tactile streams with ground truth across cucumber peeling and whiteboard wiping. The examples report close visual agreement and tactile predictions that capture contact interaction.
- Qualitative evaluation compares predicted rear- and front-camera frames with ground truth for cucumber peeling and whiteboard wiping.
- VTAM predicts tactile streams for both tasks to assess anticipation of contact dynamics, with arrows visualizing estimated force magnitude and direction.
- Cucumber peeling: 85% success (17/20 trials) in peeling produced strips longer than 10 cm in successful runs.
- Cucumber peeling: Cucumber peeling predictions closely match ground-truth rear- and front-camera observations, while tactile frames are predicted accurately.
- Whiteboard wiping: Whiteboard wiping predictions match ground truth across the wiping motion in the rear-camera view and reproduce the interaction’s visual dynamics in the front-camera view.
- Whiteboard wiping: Whiteboard wiping tactile predictions capture the interaction between the wiper and board surface.