PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework

Yifan Wu, Qin Li, Nan Min, Guojin Zhong, Haoyu Zhao, Zhiyuan Li, Houze Xu, Shengqi Xu, Xingyao Lin, Zijie Diao, Zhaoxiang Liu, Shiguo Lian, Shunlin Lu, Shihao Zhao, Ziyi Ye, Zuxuan Wu, Yu-Gang Jiang

arXiv:2610.10384v1cs.RO

TL;DR

Tactile-enabled robot policies lack a unified benchmark spanning simulation and the real world, despite touch providing physical information beyond vision. OpenViTac pairs those settings for evaluation and supports studies of tactile adaptation and sim–real co-training; OpenVTLA achieves higher reported average success rates than the strongest VLA, WAM, and VTLA baselines in both domains.

  • Problem

    Tactile-policy evaluations vary across sensors, tasks, embodiments, and protocols, and existing benchmarks lack comprehensive tactile-capability coverage and sim–real correspondence.

    Source: §1 · p. 2

  • Method

    OpenViTac pairs simulation and real-world tasks across four tactile-relevant capability dimensions, and the paper studies tactile representation, integration, and sim–real co-training for pretrained VLAs.

    Source: §1 · pp. 2–3

  • Results

    OpenVTLA achieves 68.7% average success in simulation versus 51.5%, 53.3%, and 61.2% for the strongest VLA, WAM, and VTLA baselines, and 54.6% in real-world evaluation versus 43.1%, 48.5%, and 51.9%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    OpenViTac provides a unified testbed for comparing VLA, WAM, and VTLA policies across simulation and the real world, while OpenVTLA achieves strong performance in both.

    Source: §5 · p. 9

    OpenViTac still has a sim–real gap in tactile observations due to limitations in tactile rendering, sensor modeling, and multimodal calibration.

    Source: §5 · p. 9

Abstract

from arXiv · show

Tactile feedback provides embodied agents with physical information beyond visual observations, enabling more reliable interaction with the real world. However, despite the rapid progress of vision-tactile-language-action (VTLA) policies, there remains a lack of unified benchmarks for evaluating tactile-enabled robot manipulation across simulation and the real world. To address this gap, we introduce OpenViTac, a visuo-tactile manipulation benchmark for evaluating robot policies across simulation and the real world. OpenViTac organizes contact-rich manipulation into four tactile-relevant capability dimensions and provides paired simulation-real-world settings for consistent evaluation of VLA, WAM, and VTLA policies. Building upon this benchmark, we investigate how different tactile representations and integration strategies affect the performance of pretrained VLA models. Correspondingly, we introduce OpenVTLA, a tactile augmentation framework that combines the best-performing representation and integration strategy. Furthermore, we leverage the paired benchmark setting to study sim-real co-training and analyze factors affecting cross-domain policy learning. Together, OpenViTac provides a unified platform for evaluating and advancing visuo-tactile robot manipulation.

1 Introduction

Vision-based policies lack direct access to physical states such as contact dynamics and interaction forces, while tactile-policy evaluations remain difficult to compare across differing tasks, sensors, embodiments, and protocols. OpenViTac provides a paired simulation–real-world benchmark, and the paper studies tactile adaptation for pretrained VLAs and sim–real co-training.

1 Introduction

  • Existing policy studies vary in sensors, tasks, robot embodiments, and protocols, while prior benchmarks lack comprehensive tactile-capability coverage, diverse hardware, and sim–real correspondence.General manipulation benchmarks primarily focus on vision-based policies.Source: §1 · p. 2
  • OpenVTLA reaches 68.7% average success in simulation, versus 51.5%, 53.3%, and 61.2% for the strongest VLA, WAM, and VTLA baselines; in real-world evaluation it reaches 54.6% versus 43.1%, 48.5%, and 51.9%.The baseline values are listed in VLA, WAM, and VTLA order.Source: §1 · p. 3
  • OpenViTac evaluates tactile-enabled manipulation across four capability dimensions and paired simulation–real-world settings, comparing VLA, WAM, and VTLA policies.The four dimensions are physical-property perception, fragility-aware manipulation, contact-rich manipulation, and precision manipulation.Source: §1 · pp. 2–3
  • The paper investigates tactile representation and integration strategies for pretrained VLAs, then introduces OpenVTLA as a tactile-augmentation framework.It also uses OpenViTac’s paired setting to study sim–real co-training and factors affecting cross-domain policy learning.Source: §1 · p. 3

2 Related Work

Prior work explores tactile-enabled policies and datasets, but differs in how touch is incorporated and how policies are evaluated. OpenViTac is positioned as a tactile-centric benchmark with aligned simulation–real tasks and additional real-world data across sensors.

2 Related Work

  • The related-work review covers visuo-tactile robot learning and visuo-tactile datasets and benchmarks.Figure 2 depicts the benchmark’s capability taxonomy, task-corresponding simulation construction, and configurable simulation appearance variations.Source: §2 · p. 3, Figure 2 · p. 4

2.1 Visuo-Tactile Robot Learning

  • Tactile-enabled policies use varied integration strategies, and the paper studies representative approaches under a shared π0.5 backbone because consensus on incorporating touch into pretrained VLAs is lacking.Examples include feature modulation, action-level conditioning, dedicated tactile experts, and late-stage action refinement.Source: §2.1 · p. 3

2.2 Visuo-Tactile Datasets and Benchmarks

  • Existing policy-level evaluations cover tactile sensing, but simulation benchmarks still differ substantially from real-world settings; OpenViTac aligns sim–real tasks and adds real-world data across tactile sensors.The cited work includes ManiFeel, TacO, UniVTAC, and NeoSim.Source: §2.2 · p. 4

3 OpenViTac

OpenViTac pairs simulation and real-world settings for tactile-enabled policy evaluation, and supports studies of pretrained-VLA tactile adaptation and sim–real co-training. The section describes the task suite, adaptation choices, and mixed-domain training protocol.

3 OpenViTac

  • OpenViTac provides 11 simulation tasks and 8 paired real-world tasks across four tactile-relevant capability dimensions for evaluating VLA, WAM, and VTLA policies.The benchmark also supports studies of pretrained-VLA tactile adaptation and sim–real co-training.Source: §3 · p. 4

3.1 Benchmark

  • The benchmark groups tasks into property perception, fragility-aware, contact-rich, and precision manipulation, assessing physical properties, delicate-object handling, sustained contact, and tight-tolerance alignment.Tasks include weight, hardness, and roughness classification; fragile-chip grasping; gear assembly and drawer pulling; and USB or block insertion.Source: §3.1 · pp. 4–5

3.2 ADAPTING PRETRAINED VLAS TO TOUCH

  • Tactile adaptation varies representation and integration for a shared pretrained π0.5 backbone; OpenVTLA uses AnyTouch2 tokens prepended to visual tokens without adding a fusion module or tactile-specific expert.Representations include SigLIP, T3, Sparsh, and AnyTouch2; integration strategies include token concatenation, feature modulation, action conditioning, a tactile-driven expert, and late refinement.Source: §3.2 · pp. 5–6

3.3 Simulation–Real-World Co-training

  • Sim–real co-training keeps real demonstrations fixed and adds successful simulated demonstrations, preserving task structure and aligning action and proprioceptive representations, coordinates, units, gripper conventions, and control frequency.The mixed-domain policies retain OpenVTLA and train jointly on both datasets; training uses a weighted behavior-cloning loss, needs neither paired cross-domain trajectories nor explicit feature alignment, and compares policies under matched real data, initialization, architecture, and optimization budget.Source: §3.3 · p. 6

4 Experiments

OpenViTac evaluates VLA, WAM, and VTLA policies across simulation and the real world, and the experiments compare tactile adaptation strategies and sim–real co-training. OpenVTLA leads the benchmark in both domains, while paired evaluation shows strong agreement between simulation and real-world policy performance.

4.1 Experimental Setup

  • The setup benchmarks six VLA models, three WAMs, FTP-1, and OpenVTLA in single-task policies, with separate variants evaluated separately and task-level success rates averaged without weighting.Implementation and training details are provided in Appendix C.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 68.7%: OpenVTLA achieves the highest simulation benchmark average, exceeding FTP-1 by 7.5 and the strongest VLA baseline by 17.2 percentage points.The comparison spans VLA, WAM, and VTLA models on OpenViTac.Source: Table 2 · p. 7, §4.2 · p. 7
  • 54.6%: OpenVTLA achieves the highest real-world average, while FTP-1 reaches 51.9% and VLA baselines range from 34.0% to 43.1%.FastWAM obtains 29.0%; tactile-enabled policies show their largest improvements on PP tasks, while PR remains challenging.Source: Table 3 · p. 7, §4.2 · p. 7
  • Across seven methods on eight shared tasks, simulation and real-world scores show strong agreement (Pearson r=0.967; MMRV=0.006), with OpenVTLA and FTP-1 ranking first and second in both.The authors assess whether simulation preserves relative policy performance, not identical success rates.Source: §4.2 · pp. 7–8

4.3 Understanding Tactile Adaptation in Pretrained VLAs

  • Token-level concatenation performs best on average and leads on PP, FA, and CR; among AnyTouch2 variants, VLM-input integration outperforms action-level conditioning and late refinement.The authors propose a pretrained temporally-aware tactile encoder whose tokens are injected directly into the VLM input sequence.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Simulation–Real-World Co-training

  • With 500 simulated demonstrations, 𝜋0.5 improves from 15% to 40% on In-USB and 35% to 50% on GA; OpenVTLA improves from 20% to 40% and 50% to 60%, respectively.Each setting uses 100 real demonstrations per task; the authors note residual cross-domain differences in contact-induced tactile signals as a possible factor in OpenVTLA’s smaller average gain.Source: §4.4 · p. 9, Table 5 · p. 9
  • Under perturbed lighting, co-training raises 𝜋0.5 success from 0% to 20% on In-USB and 10% to 40% on GA; OpenVTLA co-trained scores are 30% and 60%.OpenVTLA’s GA success remains 50% without co-training under both standard and perturbed lighting.Source: §4.4 · p. 9

5 Conclusion, Limitations and Future Work

The paper concludes that OpenViTac provides a unified benchmark for comparing tactile-enabled policies across simulation and the real world, and that OpenVTLA performs strongly in both. It also identifies limitations in tactile sim–real fidelity and outlines expansions to tasks and robot embodiments.

5 Conclusion, Limitations and Future Work

  • OpenViTac provides a unified testbed across four tactile-relevant capability dimensions for comparing VLA, WAM, and VTLA policies in simulation and the real world.The paper also reports strong OpenVTLA performance in both domains and improved real-world manipulation from additional simulated experience.Source: §5 · p. 9
  • Tactile observations retain a sim–real gap due to limitations in tactile rendering, sensor modeling, and multimodal calibration.Planned extensions include richer tactile interactions, long-horizon household tasks, and more diverse embodiments such as dexterous hands.Source: §5 · p. 9

A.1 Tactile Representation Learning

  • Tactile representation research has moved from paired visual–tactile features toward transferable, sensor-aware, and temporally-aware representations.The cited examples include UniTouch, Sparsh, UniT, T3, and AnyTouch2; the paper compares generic visual, tactile-specific, and temporally-aware representations under a common policy interface.Source: §A.1 · p. 13

A.2 Simulation and Real-World Robot Learning

  • Prior simulation benchmarks often lack direct real-world task counterparts, while alignment work emphasizes task-relevant correspondence and sim–real co-training.OpenViTac builds aligned simulated and physical tactile-manipulation tasks to study whether simulated demonstrations improve real-world visuo-tactile policy learning.Source: §A.2 · p. 13

B.1 Benchmark Scope and Task Taxonomy

OpenViTac organizes visuo-tactile manipulation into four capability categories and pairs simulated tasks with corresponding real-world instantiations. The environments share task semantics and support multiple sensors, grippers, and policy interfaces.

B.1 Benchmark Scope and Task Taxonomy

  • The benchmark covers 11 visuo-tactile manipulation tasks organized around physical-property perception, fragility-aware, contact-rich, and precision capabilities.Simulation and real-world evaluations use the same task semantics and success criteria for tasks instantiated in both domains.Source: §B.1 · pp. 13–14, Table S1 · p. 14

B.2 Simulation and Real-world Environment

  • OpenViTac builds scalable simulation environments with corresponding real-world tasks using the same robot embodiments, grippers, tactile sensors, and objectives.Simulation uses Isaac Sim, Isaac Lab, and TacEx-based tactile rendering; supported sensors include GelSight, Xense, and NeoteAI, with Franka Hand and Robotiq grippers.Source: §B.2 · p. 14
  • Standardized state and action interfaces support end-effector- and joint-space representations, including several rotation parameterizations, in both domains.The interfaces are intended to accommodate pretrained manipulation policies with different state and action formulations.Source: §B.2 · pp. 14–15

B.3 Data Collection Statistics

OpenViTac collects simulated trajectories and real-world demonstrations across tactile sensor configurations, with multiple object or language-conditioned variants used to increase task diversity.

B.3 Data Collection Statistics

  • Per tactile sensor configuration, simulation contains 1,300 episodes across 11 tasks (about 918K frames), while the real-world dataset contains 2,400 demonstrations across 8 tasks (about 2.1M frames).The statistics are reported separately for each sensor configuration.Source: §B.3 · p. 15, Table S2 · p. 15
  • Several tasks include multiple object configurations or language-conditioned variations, and simulation classification tasks balance binary physical-property conditions.Insert Block v2 has six real-world data configurations from three object semantics and two scene layouts.Source: §B.3 · p. 15

B.4 Full Task Catalog

OpenViTac catalogs 11 manipulation tasks, each designed around a tactile-relevant capability and paired across simulation and real-world settings. Success conditions reflect task completion requirements.

B.4 Full Task Catalog

  • The catalog covers 11 manipulation tasks, each evaluating a specific tactile-relevant capability.The catalog includes task objectives and high-level success conditions.Source: §B.4 · p. 15, Table S3 · p. 16
  • Task objectives are consistent across simulation and real-world settings.Source: §B.4 · p. 15
  • Success conditions are based on task completion, including object states, interaction outcomes, and physical constraints when applicable.Source: §B.4 · p. 15

C Policy Implementation and Training Details

The authors train a separate policy for each task, with demonstration counts differing for Insert Block v2. Policy-specific training recipes are retained, and cross-family comparisons reflect complete policies rather than isolated training choices.

C Policy Implementation and Training Details

  • Each task has a separately trained policy, using 100 demonstration episodes except Insert Block v2, which uses 300.The real-world episodes are randomly selected from a larger collected demonstration pool.Source: C Policy Implementation and Training… · p. 17, Table S4 · p. 17
  • Previously published policies retain their prescribed initialization and official training configurations, with specified training-step and global-batch-size adaptations.OpenVTLA uses the same implementation as π0.5; policy-specific preprocessing and action representations follow the corresponding source recipe, with adapters for benchmark training and execution.Source: C Policy Implementation and Training… · p. 17
  • Cross-family results compare complete policies, and each run is evaluated at its final checkpoint rather than a validation-selected checkpoint.Architectures and training recipes differ across policy families.Source: C Policy Implementation and Training… · p. 17

D Additional Experimental Results

Additional experiments examine tactile-specific pretraining and temporal context while holding the OpenVTLA architecture and token-level integration interface fixed. The supplement also visualizes the same insert-USB task across tactile sensors.

D Additional Experimental Results

  • The pretraining and temporal-context variants retain the same OpenVTLA architecture and token-level integration interface, changing only the factor under study.Source: D Additional Experimental Results · p. 17
  • Removing tactile pretraining lowers overall performance from 68.7% to 56.5%, with degradation across all four capability groups.The largest drop occurs on Fragility-Aware tasks, with substantial decreases on Property Perception and Contact-Rich tasks.Source: Table S5 · p. 17, D Additional Experimental Results · p. 17
  • Replacing tactile history with repeated copies of the current frame lowers overall performance from 68.7% to 58.4%.Degradation is most pronounced on Property Perception and Fragility-Aware tasks; similar temporal-context trends appear with SigLIP and T3.Source: D Additional Experimental Results · pp. 17–18, Table S6 · p. 18

E.1 Visualizations Across Tactile Sensors

  • The insert-USB visualization compares GelSight, Xense, and NeoteAI readings alongside camera views at selected interaction stages.The panels illustrate how the same task objective appears through different tactile sensing modalities.Source: §E.1 · p. 19, Figure S1 · p. 19

E.2 Simulated Tasks

The supplement visualizes simulated and real-world benchmark tasks using camera and bilateral tactile views across interaction stages. The real-world examples add qualitative context for tasks performed with physical objects and sensors.

E.2 Simulated Tasks

  • Simulated-task panels arrange main-camera, wrist-camera, and bilateral tactile observations in temporal order across all 11 tasks.Multi-case figures also show distinct task conditions or object variants.Source: §E.2 · p. 20
  • The simulated visualizations include physical-property perception, fragility-aware manipulation, contact-rich manipulation, and precision manipulation.Source: §E.2 · p. 20

E.3 Real-World Tasks

  • Real-world panels show camera and fingertip tactile views across successive stages of paired benchmark tasks.Examples provide qualitative context for object selection, gentle grasping, sustained contact, and precise insertion with physical objects and sensors.Source: §E.3 · p. 27

Related papers

Loading 2610.10384v1…