Source-linked AI summary
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
Baijun Chen, Weijie Wan, Tianxing Chen, Xianda Guo, Congsheng Xu, Yuanyang Qi, Haojie Zhang, Longyan Wu, Tianling Xu, Zixuan Li, Yizhe Wu, Rui Li, Xiaokang Yang, Ping Luo, Wei Sui, Yao Mu
TL;DR
Contact-rich manipulation needs tactile information, but large-scale tactile data and unified evaluation are difficult to obtain. UniVTAC addresses this with simulation-based data synthesis, a tactile-centric encoder, and an eight-task benchmark, with reported gains in simulation and real-world experiments.
Problem
Large-scale reliable tactile data are costly to collect, while the lack of unified benchmarks limits systematic evaluation of tactile-driven manipulation.
Method
UniVTAC synthesizes scalable visuo-tactile data for three sensors, trains an encoder with multi-pathway supervision, and evaluates policies on eight tactile manipulation tasks.
Results
48.0% average success rate with ACT and the UniVTAC Encoder exceeds VITaL’s 40.5% on the UniVTAC Benchmark.
Takeaways & Limitations
The benchmark and encoder support systematic analysis of tactile-dependent manipulation and provide representations that capture shape, contact deformation, and pose information.
Abstract
from arXiv · showhide
Robotic manipulation has seen rapid progress with vision-language-action (VLA) policies. However, visuo-tactile perception is critical for contact-rich manipulation, as tasks such as insertion are difficult to complete robustly using vision alone. At the same time, acquiring large-scale and reliable tactile data in the physical world remains costly and challenging, and the lack of a unified evaluation platform further limits policy learning and systematic analysis. To address these challenges, we propose UniVTAC, a simulation-based visuo-tactile data synthesis platform that supports three commonly used visuo-tactile sensors and enables scalable and controllable generation of informative contact interactions. Based on this platform, we introduce the UniVTAC Encoder, a visuo-tactile encoder trained on large-scale simulation-synthesized data with designed supervisory signals, providing tactile-centric visuo-tactile representations for downstream manipulation tasks. In addition, we present the UniVTAC Benchmark, which consists of eight representative visuo-tactile manipulation tasks for evaluating tactile-driven policies. Experimental results show that integrating the UniVTAC Encoder improves average success rates by 17.1% on the UniVTAC Benchmark, while real-world robotic experiments further demonstrate a 25% improvement in task success. Our webpage is available at https://univtac.github.io/.
I. INTRODUCTION
UniVTAC addresses limited tactile data and fragmented evaluation by combining simulation-based data synthesis, tactile-centric representation learning, and a unified benchmark. The platform supports scalable contact generation, while the benchmark and real-world experiments evaluate downstream manipulation policies.
- Vision alone is often insufficient for manipulation requiring accurate object pose and contact-state reasoning, including insertion and alignment.
- Large-scale tactile representation learning is constrained by costly hardware, nonstandardized sensors, and difficult real-world data collection.
- UniVTAC synthesizes controllable visuo-tactile interactions for three sensor types and trains a unified encoder with multiple supervision objectives.
- The encoder learns tactile-centric representations sensitive to contact boundaries, object pose, and interaction dynamics.
- The UniVTAC Benchmark contains eight tactile manipulation tasks and supports automated data generation and unified policy evaluation.
- Experiments evaluate representative policies on the benchmark and validate the learned representations through real-world robotic experiments.
II. RELATED WORK
Related work spans high-fidelity visuo-tactile simulation, tactile representation learning, and simulation-based policy data generation. UniVTAC builds on these directions while targeting richer tactile interaction modeling and safer contact-aware trajectories.
- High-fidelity visuo-tactile simulation uses physics-based approaches including IPC, FEM, and differentiable rendering to model sensor deformation and appearance.
- UniVTAC builds on TacEx by adding support for multiple sensors and automatic manipulation APIs.
- Tactile representation learning commonly uses reconstruction, explicit geometric supervision, or multimodal alignment, each emphasizing different aspects of tactile information.
- Global contrastive alignment can produce task-specific representations that require retraining for new tasks, limiting generalizability.
- Existing simulation data-generation pipelines often emphasize rigid or articulated manipulation and weakly model transient contact forces, deformation, and slippage.
- UniVTAC uses closed-loop tactile feedback and runtime validity checks to avoid sensor-destructive behavior and generate physically meaningful contact responses.
III. UNIVTAC
UniVTAC is an end-to-end simulation-driven framework covering visuo-tactile data synthesis, representation learning, and policy evaluation. It combines scalable annotated interactions with sensor modeling and tactile-reactive manipulation primitives.
- UniVTAC uses simulation to provide ground-truth physical states and scalable contact-interaction synthesis for tactile representation learning.
- The framework combines automated annotated data generation, auxiliary supervision from physical signals, downstream policy learning, and an eight-task benchmark.
- Built on TacEx, UniVTAC extends soft-body simulation to diverse sensor types and complex manipulation tasks.
- Three modeled sensors—GelSight Mini, ViTai GF225, and Xense WS—allow tactile algorithms to be tested across hardware specifications.
- Grasp and Probe APIs use tactile-reactive adaptive control, while Move and Place support precise trajectory generation.
- The control law uses real-time minimum depth, zero-contact depth, and a reaching threshold to regulate velocity and keep deformations physically realistic.
3) Tactile Representation Prerequisites:
UniVTAC’s tactile representation prerequisites target distinct physical properties and use varied contact interactions to generate informative training data. The automated pipeline produces a large interaction dataset for encoder pretraining.
- Tactile Representation Prerequisites: The encoder separates shape, contact, and other physical properties through distinct perception pathways that guide data generation and supervision.
- Tactile Representation Prerequisites: The dataset uses 14 geometric primitives, including convex and non-convex indenters, to expose the encoder to varied contact geometry.
- Tactile Representation Prerequisites: The UniVTAC Encoder is pretrained with shape reconstruction, contact deformation prediction, and object pose regression objectives.
- Tactile Representation Prerequisites: Randomized grasping depth generates interactions ranging from light touches to deep indentations, followed by small-scale rotations.
- Tactile Representation Prerequisites: Approximately 14,000 interaction frames per shape produce a total dataset of 205,826 samples.
B. UniVTAC Encoder
The UniVTAC Encoder learns tactile-centric representations through a shared encoder and multiple supervised pathways that reconstruct shape, model contact deformation, and regress object pose. Its multi-task training embeds complementary physical cues while discarding the decoders at deployment.
- Architecture: The shared encoder maps visuo-tactile observations to a latent representation decoded by pathway-specific heads during training.The heads impose complementary inductive biases on the learned representation.
- Deployment: At deployment, only the encoder is retained, so the auxiliary decoders add no inference-time overhead.Figure 2 illustrates reconstruction of marker-free images, deformation depth maps, and marker positions across contact geometries.
- Shape perception: Dual-view reconstruction captures global geometry by predicting both marker-patterned and marker-free tactile images.This separates intrinsic object appearance from sensor-specific markers and lighting.
- Contact perception: Contact supervision predicts gelpad depth and projected marker positions to model normal indentation, lateral shear, and tangential deformation.These signals encode local interaction dynamics through geometric deformation and marker displacement.
- Pose perception: Pose supervision regresses a 7-dimensional object pose comprising 3D translation and quaternion orientation for spatially consistent manipulation reasoning.The pathway grounds tactile observations in a metric spatial context.
- Training objective: The shared encoder and task-specific heads are trained end-to-end with a weighted multi-task MSE objective combining shape, contact, and pose losses.The reported balancing weights are λs = 1.0, λc = 0.5, and λp = 0.5.
C. UniVTAC Benchmark
The UniVTAC Benchmark combines automated simulation, contact-aware trajectory synthesis, and physics-based evaluation across eight tasks covering shape, pose, and contact-rich interaction. Its design introduces informative contact variation rather than relying on idealized trajectories alone.
- Unified pipeline: The benchmark integrates simulation APIs, annotated assets, and expert programs into a unified pipeline for automated data synthesis and policy evaluation.This supports scalable and reproducible experimentation.
- Trajectory synthesis: Ideal expert trajectories can be unsuitable for tactile learning because precise collision-free execution often makes contact events trivial or instantaneous.Such trajectories provide limited variation in contact perception.
- Trajectory synthesis: Randomized failures and contact-feedback corrections generate diverse tactile patterns for contact-rich skills such as insertion.The corrective process approximates coarse alignment followed by iterative contact-based correction.
- Visualization: Each benchmark task is visualized with critical execution key frames containing both visuo-tactile and standard visual observations.The figure displays one gripper side for clarity even though both fingertips provide tactile sensing during execution.
- Evaluation: Evaluation rejects trials with excessive gelpad penetration or significant relative slip, extending success criteria beyond reaching target coordinates.These constraints are intended to prevent degenerate solutions exploiting simulation artifacts.
- Task coverage: The eight tasks span pose reasoning, shape perception, and contact-rich interaction, including insertion and pull-out tasks requiring sequential contact reasoning.Representative tasks include Lift Bottle, Grasp Classify, Insert Hole, Insert Tube, Insert HDMI, and Pull Out Key.
IV. EXPERIMENT
The experiments evaluate manipulation policies, the UniVTAC Encoder, and simulation-to-real transfer using benchmark tasks and ablations. Across the benchmark, adding the encoder improves average success over vision-only ACT, while pretrained tactile representations provide useful comparison points.
- Experimental design: The evaluation studies representative policies, the UniVTAC Encoder against existing approaches, pathway contributions, and simulation-to-real transfer.ACT is the primary policy framework, with tactile representations added for comparison.
- Experimental design: Policies are trained on 50 automatically collected full trajectories per task and evaluated over 100 test rollouts.The benchmark compares ACT without tactile input, VITaL, and ACT with the UniVTAC Encoder.
- Benchmark results: 48.0% average success is achieved by ACT with the UniVTAC Encoder versus 30.9% for its vision-only counterpart across all tasks.The improvement is reported across multiple contact-sensitive insertion and pull-out scenarios.
- Benchmark results: VITaL attains strong performance on several tasks and near-perfect accuracy on Grasp Classify.The paper attributes this to explicit visuo-tactile pretraining aligned with shape-related contact cues.
- Interpretation: The benchmark spans tasks from visually solvable manipulation to scenarios that fundamentally rely on tactile feedback.This range enables analysis of how policy architectures exploit visuo-tactile representations rather than only aggregate success.
- Ablation: The pathways ablation compares contact-only, shape-only, their combination, and a from-scratch baseline under ACT.The ablation isolates contributions of the tactile perception pathways.
B. Efficacy of the UniVTAC Encoder
The UniVTAC Encoder improves visuo-tactile manipulation performance over contrastive, ablated, and from-scratch alternatives. Its strongest results come from jointly modeling shape, contact force, and relative pose.
- 48.0% average success rate with ACT exceeds VITaL's 40.5%, favoring the UniVTAC Encoder over contrastive pretraining.The comparison attributes the gap to reconstruction-based pretraining capturing more actionable, physically grounded contact information.
- 48.0% average success rate for the full model exceeds the 43.9% achieved by the Contact+Shape variant.The full model integrates geometric shape, contact force, and relative pose, while the variant omits pose perception.
- Pose perception is especially important for fine manipulation tasks requiring precise tactile feedback, including Lift Bottle.The full model's advantage is most pronounced in tasks requiring precise tactile feedback.
- Contact and Shape pathways improve performance over training from scratch by 7.9% and 7.6%, respectively.Both individual tactile pathways improve performance across most tasks.
- Joint optimization of geometric shape, contact dynamics, and pose perception outperforms modular or partial tactile processing.
C. Tactile Data Scaling for Encoder Pretraining
Scaling simulation-synthesized tactile pretraining data improves encoder effectiveness, while real-world experiments show transfer to physical manipulation. The encoder improves task success across three evaluated tasks without deployment-time fine-tuning.
- Performance increases monotonically as simulation-synthesized tactile data grows from 0 to 200k samples.The study evaluates pretraining volumes of 0, 1k, 5k, 10k, 50k, 100k, and 200k samples.
- 25% average real-world success-rate increase is observed when integrating the UniVTAC Encoder across the evaluated tasks.The evaluated tasks are Insert Tube, Insert USB, and Bottle Upright.
- 30% and 10% gains are achieved on Insert Tube and Insert USB, respectively, with encoder-augmented policies.
- Intermediate real-world frames expose evolving contact states and deformation cues used for fine-grained alignment and correction beyond vision-only perception.
- 35% improvement is achieved on Bottle Upright, extending the encoder's benefits beyond contact detection to object pose understanding.The representations were pretrained purely in simulation and transferred to the physical robot.
V. CONCLUSION
UniVTAC unifies scalable visuo-tactile data generation, tactile-centric representation learning, and benchmarking for contact-rich manipulation. Its encoder improves simulation performance and transfers effectively to real-world robotic tasks.
- UniVTAC combines high-fidelity tactile simulation, structured supervision, an encoder, and an eight-task benchmark for visuo-tactile manipulation.
- The UniVTAC Encoder captures object shape, contact deformation, and pose information through multi-pathway supervision.
- Simulation experiments show consistent manipulation improvements, while real-world evaluations confirm sim-to-real transfer from simulation-only pretraining.
APPENDIX B IMPLEMENTATION DETAILS FOR SIMULATION
The implementation uses transformer-based ACT policies with specified training settings and multimodal observations. Real-world qualitative rollouts show tactile feedback enabling correction, smoother motion, and fewer destructive interactions.
- ACT models use a four-layer encoder, seven-layer decoder, and predict 50 future actions from each observation.Fixed sine-cosine embeddings encode visual features, while tactile features use learnable positional embeddings.
- Insert Tube and Lift Bottle use third-person plus wrist-mounted views, while other tasks use only the third-person view.
- Tactile feedback corrects alignment after initial contact in Insert USB, reducing misinsertion and preventing jamming from excessive force.
- In Insert Tube, tactile-augmented policies probe gently and correct in real time under tight clearance, whereas vision-only policies may cause base shift or hole deformation.Tactile recovery remains possible for moderate errors, but large offsets fail in both settings.
- Tactile feedback improves Bottle Upright grip stability and motion smoothness, while vision-only control exhibits lifting jitter.
- Tactile sensing enhances behavioral compliance and reduces destructive interactions during real-world deployment.