PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation

Yuhang Zheng, Songen Gu, Yupeng Zheng, Weize Li, Yujie Zang, Shuai Tian, Xiang Li, Ce Hao, Chen Gao, Si Liu, Haoran Li, Yilun Chen, Shuicheng Yan, Wenchao Ding

arXiv:2603.19201v3cs.RO

TL;DR

Contact-rich manipulation is limited by insufficiently diverse aligned data and methods that do not explicitly predict contact dynamics or close the tactile-control loop. The paper introduces OmniViTac and OmniVTA, then reports consistent gains across six interaction categories, including generalization to unseen objects and configurations and robustness to disturbances.

  • Problem

    Visuo-tactile manipulation lacks large-scale, task-diverse aligned data and methods that explicitly model contact dynamics while supporting closed-loop tactile control.

  • Method

    OmniViTac provides aligned visuo-tactile-action data, while OmniVTA combines self-supervised tactile representation learning, short-horizon world modeling, contact-aware fusion, and high-frequency reflexive control.

  • Results

    OmniVTA consistently outperforms existing baselines across all six interaction categories, with advantages in unseen-object and geometric generalization and disturbance robustness.

  • Takeaways & Limitations

    Predictive visuo-tactile modeling combined with rapid tactile-feedback correction supports robust contact-rich manipulation across diverse tasks and configurations.

Abstract

from arXiv · show

Contact-rich manipulation tasks, such as wiping and assembly, require accurate perception of contact forces, friction changes, and state transitions that cannot be reliably inferred from vision alone. Despite growing interest in visuo-tactile manipulation, progress is constrained by two persistent limitations: existing datasets are small in scale and narrow in task coverage, and current methods treat tactile signals as passive observations rather than using them to model contact dynamics or enable closed-loop control explicitly. In this paper, we present \textbf{OmniViTac}, a large-scale visuo-tactile-action dataset comprising $21{,}000+$ trajectories across $86$ tasks and $100+$ objects, organized into six physics-grounded interaction patterns. Building on this dataset, we propose \textbf{OmniVTA}, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules: a self-supervised tactile encoder, a two-stream visuo-tactile world model for predicting short-horizon contact evolution, a contact-aware fusion policy for action generation, and a 60Hz reflexive controller that corrects deviations between predicted and observed tactile signals in a closed loop. Real-robot experiments across all six interaction categories show that OmniVTA outperforms existing methods and generalizes well to unseen objects and geometric configurations, confirming the value of combining predictive contact modeling with high-frequency tactile feedback for contact-rich manipulation. All data, models, and code will be made publicly available on the project website at https://mrsecant.github.io/OmniVTA.

I. INTRODUCTION

Contact-rich manipulation needs tactile information because vision alone cannot reliably capture contact forces, friction changes, and contact-state transitions. OmniViTac and OmniVTA address data and methodological gaps through large-scale aligned demonstrations, predictive contact modeling, multimodal fusion, and high-frequency tactile correction.

  • Vision alone cannot reliably infer the contact forces, friction changes, and subtle contact-state transitions required for wiping, assembly, and other contact-rich tasks.
  • Existing visuo-tactile datasets remain limited in aligned demonstration scale and task diversity, while current policies largely lack explicit predictive contact modeling and rapid tactile correction.
  • OmniViTac provides 21,879 aligned trajectories across 86 tasks and 100+ objects, organized into six physics-grounded interaction patterns with structured contact dynamics.
  • OmniVTA combines tactile representation learning, a predictive two-stream visuo-tactile world model, contact-aware fusion, and high-frequency closed-loop tactile control.
  • Real-robot experiments across six interaction categories show state-of-the-art performance, stronger generalization to unseen objects and configurations, and improved disturbance robustness.

C. Visuo-tactile Manipulation Datasets and Systems

OmniViTac extends prior visuo-tactile datasets from static paired observations or narrower manipulation collections to large-scale, fully aligned vision-touch-action trajectories. Its multimodal acquisition system combines robot and handheld embodiments for scalable, synchronized data collection.

  • Earlier tactile datasets mainly collected static paired observations, whereas newer robotic datasets added synchronized manipulation data but remained limited in scale or task coverage.
  • OmniViTac offers diverse tasks and fully aligned vision, touch, and action sequences to support large-scale world-model training and unified policy learning.
  • The dataset contains 21,879 synchronized trajectories spanning 86 tasks and 100+ objects, with RGB-D observations, high-frequency tactile sensing, and action streams.
  • A single-operator acquisition system records RGB-D observations, fingertip tactile signals, manipulation trajectories, and gripper aperture states during manipulation.
  • Timestamp-based post-processing aligns asynchronously recorded sensory streams to preserve temporal correspondence across modalities.
  • The xArm-7 captures robot-aligned demonstrations, while TacUMI enables flexible, efficient human manipulation for scalable collection.

3) Sensor Suite:

OmniViTac combines RGB-D vision with multiple tactile sensing modalities and synchronized action data to capture contact-rich manipulation across diverse environments and interaction patterns. Its taxonomy organizes behaviors by the contact mechanics that support successful execution.

  • Sensor Suite: The platform records wrist-view and third-person RGB-D observations alongside fingertip tactile sensing to capture geometry, surface interaction, and contact dynamics.
  • Interaction Patterns: The dataset spans Assembly, Cutting, Adjustment, Peeling, Wiping, and Grasping, with each category defined by its dominant tactile features and contact mechanics.
  • Sensor Suite: Four tactile sensors provide complementary signal modalities: Xense, Daimon, Tac3D-A1, and GelSight Mini.
  • Sensor Suite: Xense provides RGB tactile images at 30 Hz and 3D displacement fields at 60 Hz, while Daimon provides grayscale images at 30 Hz and displacement measurements at 60 Hz.
  • Sensor Suite: Tac3D-A1 supplies dense 3D displacement sensing at 30 Hz, and GelSight Mini records RGB images and displacement fields at 25 Hz.
  • Data Collection: The collection pipeline records multimodal streams during manipulation and validates synchronized trajectories through online and offline quality checks.

3) Tactile Pattern Analysis:

OmniViTac exhibits structured tactile patterns across contact area, force intensity, active contact duration, and latent signal organization. These distributions distinguish precision-oriented from surface-oriented manipulation and support the dataset’s physics-grounded design.

  • Latent tactile structure: t-SNE reveals physically interpretable tactile clusters, with Wiping and Peeling forming overlapping or adjacent groups because of shared contact mechanics.The embedding supports the taxonomy by linking latent tactile structure to interaction dynamics.
  • Tactile active ratio: Adjustment has the highest effective contact ratio at 0.67, whereas Cutting has the lowest at 0.27 and higher variance.Wiping and Peeling also require prolonged tactile interaction, with ratios of 0.49 and 0.41 respectively.
  • Contact area and force intensity: 66% of precision-oriented occurrences fall in the 0-10% contact-area range, reflecting localized contacts and low-magnitude force feedback.Assembly, Grasping, and Adjustment rely on delicate fingertip interactions for precise pose adjustment or damage prevention.
  • Contact area and force intensity: Surface-oriented tasks peak in the 70%-90% contact-area range, while Cutting peaks in the 40%-50% force-intensity range.These patterns indicate broad sensor engagement for frictional contact and higher force requirements for penetrating materials.
  • Overall characterization: These multidimensional statistics characterize OmniViTac as a structured, physics-grounded benchmark rather than a collection of random contacts.The benchmark is intended to support visuo-tactile perception and contact-rich policy learning.

IV. METHODOLOGY

OmniVTA uses a hierarchical slow–fast policy that combines predictive visuo-tactile planning with high-frequency tactile refinement. Its tactile representation module compresses marker displacements and reconstructs continuous deformation fields for downstream modeling and control.

  • System overview: OmniVTA decomposes contact-rich manipulation into slow planning and fast reflexive control.The slow policy plans long-horizon action chunks, while the fast policy refines actions using tactile feedback at 60 Hz.
  • Fast policy: The fast policy produces high-frequency action refinements at 60 Hz using tactile feedback, and final actions combine slow-planned and fast-refined outputs.This weighted combination supports both long-horizon planning and reactive control.
  • Slow policy: The slow policy combines a visuo-tactile world model with an adaptive fusion policy to plan from visual, tactile, and proprioceptive inputs.During inference, the world model predicts future tactile signals without generating visual observations, enabling higher-frequency rollout.
  • Tactile representation: TactileVAE encodes 3D marker displacements into compact spatio-temporal features and reconstructs deformation through an implicit continuous-field decoder.The representation is designed for efficient feature extraction and higher-frequency inference than high-resolution tactile images.
  • Encoder: The encoder uses joint temporal and spatial compression with a 3D convolutional VAE pretrained on real tactile data to learn low-dimensional, task-agnostic features.Causal temporal convolutions ensure each latent representation depends only on current and past observations during deployment.
  • Decoder: The implicit decoder predicts 3D deformation vectors from latent local features and positional encodings, yielding a continuous tactile surface deformation field.It samples spatial query points and combines interpolated latent features with positional encodings before decoding deformation.

3) Training Loss:

The visuo-tactile world model predicts coordinated visual and tactile dynamics from historical observations, while dynamic and amplitude weighting emphasizes high-activity contact regions during training.

  • Two-stream World Model: The two-stream world model separately models visual and tactile evolution while coordinating predictions under shared conditioning.It uses past frames to condition future multimodal generation through spatial-temporal diffusion transformers.
  • Two-stream World Model: The model encodes visual observations with an SD-VAE and tactile signals with the pretrained TactileVAE before joint multimodal processing.Historical observations and actions provide conditioning for the latent dynamics model.
  • Dynamic-aware Weighted Loss: The Dynamic-Aware Weighted Loss builds temporal-difference weight maps to emphasize rapidly changing tactile regions during optimization.The weighting targets high-frequency tactile patterns, local contact changes, and fine-grained temporal structure.
  • Dynamic-aware Weighted Loss: An amplitude weight map represents local contact intensity, and the dynamic and amplitude maps jointly modulate reconstruction loss.The maps are aligned to the tactile latent resolution before being applied.
  • Dynamic-aware Weighted Loss: The overall world-model objective combines diffusion, dynamic, and amplitude terms through weighted summation.λ1 and λ2 weight the dynamic and amplitude terms, respectively.

D. Adaptive Visuo-Tactile Fusion Policy

The Adaptive Visuo-Tactile Fusion Policy uses predicted contact evolution and current tactile observations to construct contact-aware representations and adaptively balance vision and touch for action planning.

  • Adaptive Visuo-Tactile Fusion Policy: The policy adaptively balances visual and tactile information according to predicted contact state for stable action planning.It contains an LTD Encoder, an adaptive fusion module, and a visuo-tactile diffusion policy.
  • Adaptive Visuo-Tactile Fusion Policy: Historical tactile concatenation alone cannot explicitly represent potential contact dynamics and their consequences.Tactile signals are largely inactive before contact and become informative during physical interaction.
  • Adaptive Visuo-Tactile Fusion Policy: The Latent Tactile Differential Encoder compares current tactile features with predicted future tactile features to capture potential contact states and interaction dynamics.Current and predicted features are spatially and temporally aggregated before fusion.
  • Adaptive Visuo-Tactile Fusion Policy: The final tactile representation concatenates current features, predicted features, and their difference channel-wise.The differential term highlights discrepancies that provide cues for potential contact events and interaction dynamics.

2) Adaptive Visuo-Tactile Fusion:

The Adaptive Visuo-Tactile Fusion Policy predicts contact likelihood, uses it to gate visual and tactile features, and generates short-horizon action chunks through conditional diffusion.

  • Adaptive modality fusion: Contact probability estimated from tactile features modulates how visual and tactile modalities are fused.A gating network uses the contact logit and tactile feature to produce normalized per-channel modality weights.
  • Adaptive modality fusion: The gating network outputs normalized weights Wt and Wv that modulate tactile and visual features, respectively.The tactile representation already encodes future tactile dynamics, so visual inputs are unnecessary for the gating network.
  • Adaptive modality fusion: Visual features use current and historical observations, while predicted tactile features capture contact dynamics without future visual prediction.This design reduces model complexity while retaining global visual context and predicted contact information.
  • Visuo-tactile diffusion policy: A conditional denoising diffusion model generates a short-horizon action chunk from fused visuo-tactile features and robot proprioception.The model progressively denoises Gaussian noise into a sequence of H coarse actions.
  • Training objective: The policy is jointly optimized with action and contact-prediction losses.The objective combines Lact with λctLbce.

E. Reflexive Latent Tactile Controller

The Reflexive Latent Tactile Controller adds high-frequency closed-loop correction to open-loop action chunks by comparing current and predicted tactile states with robot states.

  • Controller role: The RLTC receives single-frame tactile feedback, predicted tactile latents, and robot or gripper states to produce 60 Hz corrective actions.Its inputs and outputs are designed for high-frequency refinement of the planned action.
  • Tactile processing: Temporal compression is handled by repeating the current tactile frame four times before encoding it with the TactileVAE.World-model tactile predictions have lower temporal resolution and are similarly processed for compatibility.
  • Training data: The controller is trained on abnormal-recovery and normal-contact samples derived from task-specific expert tactile distributions.Observations outside the expert mean-and-standard-deviation range are treated as abnormal states.
  • Training targets: Residual targets are expert recovery actions minus slow-policy actions for abnormal samples, while normal-contact samples use zero residuals.This encourages corrections during abnormal contact and near-zero intervention during normal execution.
  • Training pipeline: The framework is trained in four stages: TactileVAE, visuo-tactile world model, adaptive fusion policy, and reflexive controller.The supplied training description also specifies separate data, optimization, and model-size reporting for these components.

2) Hardware Setup:

The evaluation compares OmniVTA and contact-rich manipulation baselines across multimodal generation, tactile prediction, and policy-control settings using diverse robotic-task objects.

  • Prediction baselines: Multimodal prediction baselines cover unified generation, joint trajectory diffusion, and conditional modality prediction.These comparisons include UVA, ForceMimic, KineDex, and exUMI formulations.
  • Contact-rich policy baselines: The policy baselines include RGB-only Diffusion Policy, PCA-augmented tactile policy, and force-conditioned diffusion methods.KineDex and ForceMimic jointly predict actions and forces, while DP and DP+tactile provide simpler comparisons.
  • Compared policies: OmniVTA combines a visuo-tactile world model with a diffusion planner and reflexive tactile controller.The open-loop ablation removes RLTC and executes diffusion-planned action chunks without reflexive feedback.
  • Task objects: The manipulation tasks use objects shown in the evaluation object set and compare both the full framework and its open-loop variant.The object inventory is presented alongside the policy comparisons.

B. Overall Performance

OmniVTA is evaluated on object diversity, geometric and tool generalization, and perturbation robustness, with results favoring the full closed-loop framework across the six task categories.

  • Generalization: Evaluation tests position generalization at unseen heights and tool generalization with an unseen knife.Position tests cover wiping, peeling, assembly, and adjustment; tool tests cover cutting.
  • Object diversity: OmniVTA achieves the best object-diversity performance across all six tasks, and closed-loop control improves over the open-loop variant.The comparison reports success rates under object-diversity, generalization, and perturbation settings.
  • Contact regulation: RDP produces excessive contacts with average deformation 0.56 and maximum deformation 1.1, whereas OmniVTA regulates motion using predicted tactile targets.The cited comparison attributes the difference to the reflexive controller's tactile regulation of contact forces.
  • Generalization: OmniVTA without RLTC already surpasses RDP under unseen-height evaluation, while replacing the training knife has little impact on cut-task performance.The results suggest robustness to geometric variation and reduced reliance on memorized demonstration trajectories.
  • Perturbation robustness: OmniVTA consistently achieves the highest perturbation success rates, and RLTC improves recovery of stable contact over open-loop execution.Perturbations vertically displace the target object during interaction and disrupt the current contact state.
  • Tactile representation: TactileVAE achieves the best reconstruction performance across all six interaction categories using L2 distance and cosine similarity.Lower L2 and higher cosine similarity indicate better reconstruction performance.
  • Tactile representation: Local feature maps outperform global-token representations in reconstruction accuracy and tactile embedding informativeness.Implicit neural representations also produce clear cross-sensor clustering of force patterns, unlike marker-based or positionless variants.

2) Visuo-Tactile World Model:

The Visuo-Tactile World Model predicts future visual and tactile signals over multiple horizons and is evaluated for accuracy, component effectiveness, and perturbation robustness.

  • Prediction accuracy: The model consistently outperforms four baselines in short-term and long-term tactile prediction.Evaluation uses L2 distance and cosine similarity at multiple latent and decoded prediction horizons.
  • Joint visual–tactile generation: Jointly generating future visual and tactile features improves tactile prediction accuracy by providing complementary global dynamic cues.
  • Action representation: 2D action conditioning achieves the best generalization to unseen object positions, followed by 3D relative actions.The result suggests that action conditioning primarily conveys motion intent and that 2D actions align naturally with visual observations.
  • Dynamic weighting: Dynamic weighting improves tactile prediction by emphasizing rapidly changing regions, strong contact responses, and fine-grained temporal structures.
  • Perturbation robustness: The model exhibits recovery capability when visual and tactile observations are replaced with non-contact observations after contact.The perturbation experiment evaluates robustness in the wipe-vase task.

3) Adaptive Visuo-Tactile Fusion Policy:

The Adaptive Visuo-Tactile Fusion Policy combines current and predicted tactile information with visual features, using gating to adjust modality reliance during contact-rich execution.

  • Predicted tactile signals: Predicted tactile information consistently improves policy success rates, with six-step horizons outperforming four- and two-step horizons.
  • Reflexive control: The framework includes a reflexive latent tactile controller that rapidly restores contact after perturbations.The controller operates in the execution loop and is evaluated by comparing recovery with and without RLTC.
  • Tactile representation: The LTD-based tactile representation achieves higher success rates across multiple tasks than simple feature concatenation.The encoder captures the dynamic relationship between current and predicted tactile signals.
  • Gating fusion mechanism: The gating mechanism improves average task success rate by approximately 7% over direct feature concatenation.Tactile weight remains near zero without contact and increases as predicted contact probability rises.
  • Tactile prediction accuracy: As tactile prediction accuracy decreases, the policy progressively loses its ability to infer contact states and adjust modality weights, reducing success rate.The controlled evaluation uses prediction accuracies corresponding to 80%, 60%, 40%, and 20% of the best-performing model.
Loading 2603.19201v3…