Source-linked AI summary
Visuo-Tactile World Models
Carolina Higuera, Sergio Arnaud, Byron Boots, Mustafa Mukadam, Francois Robert Hogan, Franziska Meier
TL;DR
Vision-only world models can misrepresent contact-rich interactions when forces are hidden or objects are occluded. VT-WM combines visual and tactile sensing to model contact dynamics, improving imagined physical fidelity and zero-shot real-robot planning, while its generalization remains bounded by the evaluated tasks and objects.
Problem
Vision-only world models struggle with hidden contact forces and occlusion, producing disappearing, teleporting, or physically implausible object motion.
Method
VT-WM combines visual and tactile observations in an action-conditioned world model and uses its imagined rollouts for CEM-based planning.
Results
VT-WM improved imagination and contact-rich planning, reducing normalized Fréchet distance by 33% on average and achieving up to 35% higher real-robot success rates.
Takeaways & Limitations
Grounding world-model imagination in contact dynamics supports more reliable zero-shot planning for pushing, wiping, and stacking than vision-only modeling.
Takeaways & Limitations
Evaluation leaves generalization to entirely novel tasks, objects, and object properties untested, and CEM planning remains computationally expensive.
Abstract
from arXiv · showhide
We introduce multi-task Visuo-Tactile World Models (VT-WM), which capture the physics of contact through touch reasoning. By complementing vision with tactile sensing, VT-WM better understands robot-object interactions in contact-rich tasks, avoiding common failure modes of vision-only models under occlusion or ambiguous contact states, such as objects disappearing, teleporting, or moving in ways that violate basic physics. Trained across a set of contact-rich manipulation tasks, VT-WM improves physical fidelity in imagination, achieving 33% better performance at maintaining object permanence and 29% better compliance with the laws of motion in autoregressive rollouts. Moreover, experiments show that grounding in contact dynamics also translates to planning. In zero-shot real-robot experiments, VT-WM achieves up to 35% higher success rates, with the largest gains in multi-step, contact-rich tasks. Finally, VT-WM demonstrates significant downstream versatility, effectively adapting its learned contact dynamics to a novel task and achieving reliable planning success with only a limited set of demonstrations.
1. Introduction
VT-WM extends robot world models beyond visual imagination by adding touch, grounding predictions in local contact signals. This improves imagined physical behavior and zero-shot planning in contact-rich manipulation.
- Motivation: VT-WM combines vision for global scene context with touch for local contact dynamics.The multimodal model helps maintain object permanence under occlusion and distinguish visually similar contact states.
- Evaluation: The evaluation targets contact-rich tasks where vision-only models can hallucinate disappearing, teleporting, or physically implausible objects.These failures arise under occlusion or visual aliasing.
- Results: 33% gain in object permanence and 29% gain in compliance with physical laws were achieved across manipulation tasks.These gains concern imagination quality during autoregressive rollouts.
- Results: 35% higher success was achieved in zero-shot real-robot planning for contact-rich tasks.The evaluation compares VT-WM with a vision-only world model.
2. Related Works
Prior robot world models largely rely on vision or task-specific touch dynamics, while multimodal vision-and-touch world models remain comparatively underexplored. This work positions VT-WM as a general-purpose, action-conditioned multimodal alternative.
- World-model foundations: The proposed model uses latent-state world modeling with an action-conditioned dynamics model.Observations are projected into latent representations before dynamics learning.
- Tactile representation learning: Recent tactile encoders learn low-dimensional representations from vision-based tactile sensors without explicit labels.Examples include Sparsh and Sparsh-X.
- Robot control: World-model research has limited evidence for using general-purpose action-conditioned video models to control robots.Many such works emphasize generation capabilities rather than robot control.
- Visual world models: Most prior world models for real-world tasks use visual dynamics models trained in pixel space, latent space, or structured state representations.The prior approaches include keypoint and tracked 3D-state representations.
- Multimodal gap: Few studies train world models jointly with vision and touch, and existing touch dynamics models are generally task-specific.VT-WM instead targets general-purpose multimodal world models for visual model-predictive control.
3. World Models that Understand Contact
VT-WM addresses failures of vision-only world models by combining exocentric vision, tactile sensing, and action-conditioned prediction. Its architecture produces multimodal latent futures that can support physically grounded imagination and planning.
- Problem: Vision-only models struggle with contact because occlusion and invisible forces can produce teleportation, disappearance, or implausible object motion.These artifacts occur during grasping, pushing, or placing.
- Approach: VT-WM uses vision and touch to generate more accurate rollouts for contact-rich manipulation.Touch supplies contact information during occlusion and grounds imagination in contact physics.
- Architecture: The architecture encodes exocentric video and tactile feedback, fuses them with control actions, and predicts next-step visual and tactile states.The predictor is an autoregressive forward dynamics model.
- Multimodal reasoning: Tactile context disambiguates visually identical states that can lead to different imagined outcomes.For example, contact can distinguish lifting a cup from leaving it on the table.
- Architecture: Factorized spatial and temporal attention captures local contact dynamics and global context while avoiding full spatiotemporal attention complexity.Action tokens undergo the same factorized attention process.
- Training: Training combines teacher-forced next-state prediction with autoregressive sampled trajectories to improve long-horizon generation.The two objectives are combined with equal weighting, and sampled states are generated without gradients.
- Planning: CEM evaluates candidate action sequences by autoregressively rolling out latent futures and scoring their distance from a goal image latent.Tactile information improves the reliability of the learned model and indirectly supports vision-based planning.
4. Experiments
Experiments evaluate whether tactile grounding improves contact perception, physical plausibility, real-robot planning, and adaptation to a novel task. VT-WM generally outperforms V-WM, with strongest gains in contact-rich and multi-step settings.
- Experimental goals: The experiments assess contact perception, zero-shot planning transfer, and downstream versatility in contact-rich manipulation.They compare visuo-tactile and vision-only world models using rollout metrics, real-robot plans, and limited demonstrations for a novel task.
- Object permanence: VT-WM preserves object representations during occlusion and reintroduces them at the correct location after release.In cube stacking, the blue cube remains represented during transport and placement, then reappears above the yellow target cube.
- Object permanence: 33%: VT-WM’s average reduction in normalized Fréchet distance for object trajectories relative to V-WM.Across five tasks, reductions range from 18–47%, with statistically significant improvements in place fruits, push fruits, and cube stacking.
- Causal compliance: 29%: VT-WM’s average reduction in hallucinated motion for passive objects, indicating stronger causal compliance across tasks.The metric uses normalized Fréchet distance for objects that should remain stationary unless acted on by external forces.
- Zero-shot planning transfer: VT-WM improves real-robot planning most in contact-rich tasks, including gains of 35% on reach & push and 31% on wipe cloth.Both models reach 100% on reach button, while VT-WM also improves push fruits by 10% and stack cubes by 11%.
5. Discussion and Conclusion
VT-WM grounds visual imagination in contact physics by integrating fingertip tactile sensing with exocentric vision. This improves object-dynamics fidelity and zero-shot planning, especially for contact-rich manipulation.
- VT-WM integrates fingertip tactile sensing with exocentric vision to ground imagination in contact physics.This complements global visual context with local contact information and addresses vision-only hallucinations such as disappearance, teleportation, and unrealistic deformation.
- 33% lower normalized Fréchet distance shows improved representation of true object dynamics relative to V-WM.VT-WM preserved object permanence under occlusion and maintained resting states when objects lacked external forces.
- Up to 35% higher success rates were achieved by VT-WM in zero-shot real-robot planning for contact-rich tasks.The gains included pushing, wiping, and stacking, while performance was similar on free-space reaching.
B.0.1. TRAINING DATASET
The training dataset consists of diverse teleoperated multimodal trajectories spanning eight contact-rich manipulation tasks. It deliberately includes both successful and failed demonstrations.
- Each sequence records proprioception, exocentric video, and video from four fingertip tactile sensors.The hardware uses a Franka Panda arm, Allegro Hand, Digit 360 fingertip sensors, and an exocentric camera.
- The dataset contains teleoperated robot-arm trajectories for eight distinct contact-rich manipulation tasks.Tasks include pick and place, button pressing, pushing, wiping, insertion, stacking, and marker scribbling.
- Each task includes both successful and failure demonstrations rather than successes alone.The trajectories were collected without discriminating between outcomes.
B.1. Training parameters
The training setup uses teleoperated multimodal trajectories and a large-scale AdamW optimization schedule. The supplied passages provide dataset context and selected optimization parameters, but not a complete training-parameter specification.
- Training trajectories were collected by teleoperation and include successful and failure sequences.The dataset passage describes the multimodal trajectories used to train the world model.
- AdamW training used β1 = 0.9, β2 = 0.95, weight decay 0.01, and an effective batch size of 64 on 32 A100 GPUs.The schedule used 80,000 updates with linear warmup followed by cosine decay.
- The learning rate warmed up to 3e-4 and decayed to 3e-7.
C. Contact Perception with Visuo-Tactile World Model
VT-WM produces action-conditioned visuo-tactile rollouts that align with commanded robot motions and capture visual outcomes and contact events. These predictions support its use as a unified predictive framework for planning.
- VT-WM rollouts align with commanded translations along ±x, ±y, and ±z and with hand opening or closing.The model receives proprioceptive state deltas and produces coherent directional and finger-configuration changes.
- The learned dynamics produce coherent action-conditioned behavior without explicit supervision for axis-aligned motion.
- Held-out real robot action sequences from button-pressing and marker-scribbling tasks were used to evaluate autoregressive predictions.
- Predicted visual states closely resemble final RGB observations, while predicted tactile states capture plausible contact events and finger-object interactions.In the table-leg insertion rollout, predicted contact signatures match the fingers in contact in the ground-truth sequence.
- Consistent visuo-tactile rollouts represent global visual context and local contact dynamics in one predictive framework for planning.
D.1. CEM algorithm for planning with World Models
CEM plans robot actions by rolling candidate sequences through a world model and selecting those whose predicted outcomes best match the goal. Across contact-rich tasks, VT-WM’s contact-aware imagination supports more reliable real-robot execution than V-WM.
- CEM planning: CEM encodes the goal and current vision-tactile context, samples action sequences, rolls them out in the world model, and evaluates predicted outcomes against the goal.The algorithm iteratively updates its action distribution using the lowest-cost particles before executing the best sequence.
- CEM planning: VT-WM predictions match ground-truth tactile contact signatures for the fingers during cube-stacking rollouts.The rollout comparison includes ground-truth vision, predicted vision, ground-truth tactile signatures, predicted tactile signals, and their difference from no contact.
- Reach Button: Both V-WM and VT-WM generate feasible Reach Button trajectories because the task mainly tests spatial reasoning and kinematic alignment.The comparison covers imagined CEM plans and their zero-shot executions on the real robot.
- Push Fruits: VT-WM preserves object geometry during Push Fruits imagination, while V-WM distortions can cause the object to topple instead of sliding.The V-WM deployment also produces shorter object displacement, whereas VT-WM supports stable sliding.
- Reach & Push: VT-WM establishes contact during Reach & Push, enabling successful completion of both reach and push subgoals, unlike V-WM.V-WM’s hand hovers above the object, so its subsequent push occurs without contact and fails to move it.
- Wipe Cloth: VT-WM maintains contact and clearer geometry in Wipe Cloth, producing stable horizontal cloth displacement, while V-WM frequently fails to move the cloth.V-WM rollouts show hovering and geometric artifacts associated with ineffective wiping during execution.
- Stack Cubes: VT-WM consistently represents the cube through transport and placement, enabling stable zero-shot stacking, whereas V-WM’s cube can disappear and fail placement.The V-WM artifact reflects tracking hand–scene geometry without a consistent hand–object relationship.
E. Limitations
The evaluation and planning setup has several scope and deployment boundaries. These include dependence on a specific tactile sensor, limited task and object generalization, computationally expensive CEM planning, and open-loop execution chunks.
- Sensor scope: The tactile modality is limited to vision-based tactile sensing, specifically the Digit 360 sensor.The framework may extend to other tactile modalities when an appropriate pretrained tactile encoder is available.
- Generalization: Contact-perception evaluation uses unseen robot trajectories only within tasks from the training distribution, leaving generalization to entirely novel tasks or object characteristics open.
- Generalization: Planning tests use the same scene and objects, without evaluating objects with different visual or physical properties such as size, shape, or color.
- Planning deployment: CEM planning is computationally expensive because it requires many autoregressive rollouts per particle and therefore executes trajectory chunks open-loop.This differs from classical policies that operate closed-loop at higher control frequencies.