Source-linked AI summary
ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
Yunao Huang, Shiyu Sang, Suting Ni, Haotao Lu, Shijie Wu, Rui Lv, Ziyang Guo, Ye Shi, Jingya Wang
TL;DR
Scaling synchronized visuo-tactile-action trajectories is difficult because real tactile interaction data are costly to collect and simulated data have fidelity, diversity, and transfer constraints. ViTacWorld predicts temporally aligned visual and tactile rollouts conditioned on robot actions, pretrained on public and simulated data and finetuned with real policy rollouts. Tactile π0.5 mean success rises from 43.0% (expert demonstrations) to 67.5% and 79.5% after successive rounds; ACT + tactile rises from 17.0% to 47.0%; imagined evaluations correlate (Pearson r = 0.863).
Problem
Scaling synchronized visuo-tactile-action trajectories is difficult because real tactile interaction data are costly to collect and simulated data have fidelity, diversity, and transfer constraints.
Source: §I · p. 1Method
ViTacWorld predicts temporally aligned visual and tactile rollouts conditioned on robot actions, pretrained on public and simulated data and finetuned with real policy rollouts.
Source: §I · p. 2Results
Tactile π0.5 mean success rises from 43.0% (expert demonstrations) to 67.5% and 79.5% after successive rounds; ACT + tactile rises from 17.0% to 47.0%; imagined evaluations correlate (Pearson r = 0.863).
Source: B. Policy Improvement with Generated… · p. 5, C. Policy Evaluation with Imagined… · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/
I. INTRODUCTION
Contact-rich learning needs tactile cues, but real interaction data are costly and simulations remain limited in fidelity and diversity. ViTacWorld extends action-conditioned world modeling to aligned visual-tactile rollouts, trained on public and simulated data and adapted with real policy rollouts for policy improvement and evaluation.
I. INTRODUCTION
- Real tactile demonstrations are costly and slow to collect, while simulation is constrained by task diversity, asset and contact fidelity, and sim-to-real transfer.Tactile sensors can have limited lifetimes under repeated contact, and some deformable or frictional tasks remain difficult to reproduce faithfully.Source: §I · p. 1
- Action-conditioned world models can generate contact-rich trajectories by predicting how robot actions jointly evolve visual observations and tactile feedback.The paper frames this as extending world modeling beyond vision-only prediction toward reusable data generation for downstream tactile policies.Source: §I · p. 2
- ViTacWorld generates temporally aligned visual and tactile rollouts conditioned on robot action chunks, using view-aware conditioning and cross-view attention.It is pretrained on large-scale public and simulated data, then finetuned on real demonstrations and target-setup policy rollouts.Source: §I · p. 2
- The training pipeline combines public visuo-tactile-action data, simulated interactions, and real-world policy-rollout finetuning to scale visuo-tactile world modeling.The paper presents the framework as the first world-model-based approach to robot visuo-tactile-action trajectory generation.Source: §I · p. 2
- Generated rollouts support both downstream tactile-policy improvement and action-conditioned policy evaluation under controlled action sequences.Evaluation predicts visuo-tactile outcomes; policy improvement uses synthetic rollouts as training data.Source: §I · p. 2
II. RELATED WORK
Prior work incorporates tactile feedback into policies and provides collection or simulation routes for visuo-tactile data, but these approaches retain scaling constraints. ViTacWorld instead extends an action-conditioned robot video prior into a data generator for downstream tactile policies.
II. RELATED WORK
- Visuo-tactile policies and tactile-augmented VLA models use tactile feedback for contact-sensitive control, whereas ViTacWorld focuses on generating additional data to train those policies.The paper presents data generation as complementary to integrating tactile observations or predictions into policies.Source: §II · p. 2
- Real collection still requires tactile hardware, synchronization, and repeated contact, while simulated data remain limited by contact fidelity, asset realism, task diversity, and sim-to-real transfer.Robot-free and handheld collection can also introduce embodiment or viewpoint gaps.Source: §II · p. 2
- ViTacWorld extends a large action-conditioned robot video prior into a generator of aligned visual-tactile rollouts for downstream tactile-policy learning.Prior tactile world-modeling work explored contact-dynamics consistency but was not primarily designed for scalable policy-training data generation.Source: §II · p. 3
III. METHOD
- Figure 2 summarizes ViTacWorld’s architecture, two-stage training pipeline, and two downstream uses.The passage identifies the figure’s scope but does not specify its visual encodings.Source: §III · p. 3
A. Problem Formulation
ViTacWorld predicts future visual and tactile observations from current multimodal observations and robot action sequences. It adapts a pretrained action-conditioned video model with stream-aware processing and cross-view exchange, training on future valid streams for aligned rollouts.
A. Problem Formulation
- The model takes main-camera, wrist-camera, and tactile observations plus an H-step sequence of relative end-effector motions and gripper commands, and predicts future visual and tactile observations.A view-presence mask indicates which streams are available in heterogeneous training data.Source: A. Problem Formulation · p. 3
B. Action-Conditioned Visuo-Tactile World Model
- ViTacWorld retains a pretrained action-conditioned robot video model’s backbone and action pathway, encoding camera and tactile streams separately before assembling their latent tokens.The approach reuses robot motion priors while adapting the model to generate aligned visual and tactile observations.Source: B. Action-Conditioned Visuo-Tactile… · p. 3
- Stream-aware self-attention processes tokens within each stream, while explicit cross-view attention exchanges information across streams.Stream identity embeddings enter the timestep-and-action modulation path.Source: B. Action-Conditioned Visuo-Tactile… · p. 3
- Separating ordinary self-attention from cross-view exchange avoids uncontrolled mixing of camera and tactile tokens while allowing contact information to pass between streams.Tactile is generated as a first-class stream temporally aligned with the visual rollout.Source: B. Action-Conditioned Visuo-Tactile… · p. 3
- Robot action sequences are embedded through the inherited pathway and repeated across visual and tactile streams at corresponding latent timesteps.The original timestep/AdaLN modulation path injects the action conditioning.Source: B. Action-Conditioned Visuo-Tactile… · p. 3
- Training uses the video world model’s latent denoising objective, predicting future visual and tactile latents conditioned on the current observation, actions, stream identities, and view-presence mask.The loss applies only to future prediction frames and valid streams.Source: B. Action-Conditioned Visuo-Tactile… · p. 3
C. ViTacWorld Pretraining and Fine-Tuning
ViTacWorld is pretrained on large-scale real and simulated visuo-tactile-action trajectories, then tuned with real demonstrations and policy rollouts. Task-aligned simulation adds contact-pattern diversity, while real rollout data adapt the model to target-policy states.
Pretraining with large-scale real and simulated data
- Pretraining combines large-scale public and simulated visuo-tactile data, using view-presence masks so incomplete trajectories can contribute without corrupting missing streams.The first stage learns broad action-conditioned visual-tactile dynamics before target-domain adaptation.Source: C. ViTacWorld Pretraining and… · p. 3
- Task-aligned simulation uses reconstructed target scenes, calibrated camera extrinsics, and reproduced camera-robot layout, while varying assets and interaction instances for added object and task diversity.Tactile rendering uses the Xense pipeline in Isaac Sim; the authors note that tactile appearance can have a relatively smaller modality gap when sensor geometry and rendering are aligned.Source: C. ViTacWorld Pretraining and… · pp. 3–4
C. ViTacWorld Pretraining and Fine-Tuning
- The two-stage pipeline uses real and task-aligned simulated data for pretraining, followed by real robot task data and policy rollouts for target-domain tuning.The overview depicts these stages as the scaling training pipeline.Source: Figure 2 · p. 4
Pretraining with large-scale real and simulated data
- Stage-I pretraining mixes over 16K public real trajectories with over 5K task-aligned simulated trajectories at an approximately 2:1 sampling ratio.Training uses 13-frame windows sampled at 15 Hz for 30K steps.Source: C. ViTacWorld Pretraining and… · p. 4
Real-world policy rollout and finetuning
- Stage-II tuning uses real expert demonstrations and policy rollouts, exposing the model to downstream-policy states from both successful and failed contact-rich interactions.This stage is intended to better match the deployment distribution for later synthetic rollout generation.Source: C. ViTacWorld Pretraining and… · p. 4
D. Visuo-Tactile Policy Improvement and Evaluation
ViTacWorld uses downstream tactile-policy action chunks to autoregressively generate imagined visual-tactile rollouts. The rollouts can be screened for augmentation data or inspected as a lightweight policy-evaluation signal before deployment.
D. Visuo-Tactile Policy Improvement and Evaluation
- Given real initial observations and action chunks from a downstream tactile policy, ViTacWorld generates contact-rich visual-tactile rollouts autoregressively.Policy actions can be converted to the world-model action space before predicting an H-step future rollout.Source: D. Visuo-Tactile Policy Improvement… · p. 4
- Selected successful imagined rollouts are converted into dream data to augment real demonstrations for downstream tactile-policy training.The augmented dataset can be used to finetune the policy.Source: D. Visuo-Tactile Policy Improvement… · pp. 4–5
- Each generated observation becomes the next policy input, creating a receding-horizon rollout that reflects states likely to be visited by that policy.Generated rollouts are filtered for task success and visual-tactile plausibility before selected trajectories are merged with expert demonstrations.Source: D. Visuo-Tactile Policy Improvement… · p. 4
- The same policy-in-the-loop imagined executions can be inspected as a lightweight evaluation signal before real-robot deployment.Rollouts can be inspected for success or failure.Source: D. Visuo-Tactile Policy Improvement… · pp. 4–5
IV. EXPERIMENTS
The experiments assess ViTacWorld’s generated rollouts, downstream policy use, and the effects of training choices. They examine policy improvement, preservation of real-robot performance in imagined executions, and generation quality.
IV. EXPERIMENTS
- The experiments test whether ViTacWorld-generated rollouts improve contact-rich manipulation policies.They also examine whether imagined executions preserve real-robot policy performance and how training choices affect generation quality.Source: §IV · p. 5
A. Experimental Setup
The experiments use a tactile-equipped Franka Panda on four contact-rich tasks, with expert demonstrations and policy-generated rollouts supporting training and evaluation across policy stages.
A. Experimental Setup
- Experiments use a Franka Panda with a Robotiq gripper, finger-mounted Xense tactile sensors, and external and wrist-mounted RGB cameras.The tactile policies also receive proprioception and task instructions; expert demonstrations are collected through FACTR teleoperation.Source: A. Experimental Setup · p. 5
- The four tasks are Charger Plugging, Cucumber Peeling, U-Block Insertion, and Cuboid Insertion, spanning precise alignment, peeling contact, and constrained insertion.Charger Plugging is particularly challenging because small pose errors can cause insertion failure despite a similar-looking visual trajectory.Source: A. Experimental Setup · p. 5
- The study collects 300 expert demonstrations and 50 real policy rollouts per task to tune ViTacWorld to downstream-policy state distributions.The policy rollouts include successful and failed executions.Source: A. Experimental Setup · p. 5
- From candidate trajectories, 200 successful rollouts are selected across tasks after human checks of task completion and visual-tactile plausibility.Task-specific checks include complete insertion and continuous valid tool-object contact.Source: A. Experimental Setup · p. 5
- The study compares vision-only π0.5, tactile π0.5, and ACT + tactile policies at expert-only, round-1, and round-2 training stages.Round 2 adds 200 rollouts to the retained 200 round-1 rollouts and 300 demonstrations, totaling 700 trajectories; tactile π0.5 generates the rollouts used to finetune all policies.Source: A. Experimental Setup · p. 5
- Real-robot success rates are measured over 50 trials per policy, stage, and task, with success defined by completion of the task-specific goal.Figure 3 qualitatively contrasts the same π0.5 + tactile architecture trained with expert demonstrations alone versus demonstrations augmented by generated rollouts.Source: A. Experimental Setup · p. 5, Figure 3 · p. 5
B. Policy Improvement with Generated Visuo-Tactile Data
Generated visuo-tactile rollouts improve real-robot policy performance across tasks and policy architectures. Comparisons also show gains over matched visual-action augmentation and illustrate changes in grasping and contact adjustment.
B. Policy Improvement with Generated Visuo-Tactile Data
- With rollout counts and visual-action sequences matched, tactile π0.5 gained 24.5 versus 20.5 percentage points after round 1, and 36.5 versus 34.5 after round 2, compared with vision-only.The tactile policy also maintained higher success on every task at every training stage.Source: B. Policy Improvement with Generated… · p. 5
- After augmentation, the tactile policy showed more stable grasping and contact-phase corrections, including small re-alignments after an initially misaligned insertion attempt.Figure 3 compares real-robot rollouts for the same policy architecture with expert-only training versus generated-rollout augmentation.Source: B. Policy Improvement with Generated… · p. 5
- Mean success rate for tactile π0.5 rose from 43.0% with expert demonstrations to 67.5% and 79.5% after successive augmentation rounds; ACT + tactile rose from 17.0% to 47.0%.The table reports real-robot success rates over 50 trials per task, policy, and training stage, with the average taken across four tasks.Source: B. Policy Improvement with Generated… · p. 5, Table I · p. 6
C. Policy Evaluation with Imagined Rollouts
The evaluation compares imagined and real executions from matched initial states across tasks and checkpoints. Imagined rates track relative checkpoint ordering, while a matched qualitative example illustrates agreement on a task-level failure.
C. Policy Evaluation with Imagined Rollouts
- Across 12 checkpoint-task rate pairs, real and imagined performance were positively associated (Pearson r = 0.863; Spearman ρ = 0.790), and both ranked checkpoints expert-only < round 1 < round 2 within every task.Imagined success was underestimated by 9.0 points on average, including a 26-point error for round-2 Cucumber Peeling.Source: C. Policy Evaluation with Imagined… · p. 6
- Aggregate-rate comparisons support relative checkpoint comparison, but do not establish exact success-rate calibration or trial-level outcome agreement.For each checkpoint-task combination, 50 real-imagined pairs were initialized from recorded trial states and observations, totaling 600 pairs; the same fixed task-completion criteria were applied.Source: C. Policy Evaluation with Imagined… · p. 6
- In a matched U-Block example, the real execution and all three independently sampled imagined rollouts fail, despite trajectories that need not be frame-identical.This qualitative example is separate from the 50-trial quantitative evaluation.Source: C. Policy Evaluation with Imagined… · p. 6, Figure 5 · p. 6
D. Visuo-Tactile World Model Quality Analysis
Pretraining improves world-model prediction quality over direct real-data tuning, and task-aligned simulation further improves it. Cross-view attention supports perceptual prediction quality across visual and tactile streams, with qualitative rollouts showing clearer geometry and fewer hallucinated contact states.
D. Visuo-Tactile World Model Quality Analysis
- Evaluation uses the first observation and logged actions to predict future frames, scoring each view with PSNR, SSIM, and LPIPS against ground truth.All variants are evaluated on the same held-out real-world clips spanning contact transitions.Source: D. Visuo-Tactile World Model Quality… · pp. 6–7
- Replacing cross-view attention with self-attention raises LPIPS in the main view, wrist view, and tactile stream, while main-view SSIM is nearly unchanged.LPIPS changes from 0.0513 to 0.0611 in the main view, 0.0725 to 0.0854 in the wrist view, and 0.0157 to 0.0163 in the tactile stream; main-view SSIM is 0.8286 versus 0.8290.Source: D. Visuo-Tactile World Model Quality… · p. 7
- On held-out validation clips, the full model produces clearer rollouts, better-preserved object geometry, more consistent end-effector motion, and fewer hallucinated contact states.Figure 6 presents qualitative predictions across visual and tactile views.Source: D. Visuo-Tactile World Model Quality… · p. 7, Figure 6 · p. 7
- Pretraining improves generation quality over direct real-data tuning, and task-aligned simulated tactile trajectories further improve the model.Table II reports prediction quality on held-out contact-transition clips, but the supplied text gives no metric values.Source: Table II · p. 7, D. Visuo-Tactile World Model Quality… · p. 7
V. CONCLUSION
ViTacWorld generates temporally aligned visual and tactile rollouts under robot actions, using real and simulated trajectories followed by real-world policy-rollout fine-tuning. Experiments on contact-rich tasks show policy-improvement rollouts and imagination-based policy evaluation.
V. CONCLUSION
- ViTacWorld generates temporally aligned visual and tactile rollouts under robot actions for contact-rich manipulation.The model is trained on large-scale real and simulated trajectories, then fine-tuned with real-world policy rollouts.Source: §V · p. 7
- The generated rollouts serve as synthetic visuo-tactile-action data and as a lightweight tool for policy evaluation.Source: §V · p. 7
- Experiments on contact-rich tasks show that ViTacWorld produces policy-improvement rollouts and supports imagination-based policy evaluation.Source: §V · p. 7