PaperlayerView exact revision on arXiv

Source-linked AI summary

ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning

Quan Khanh Luu, Pokuang Zhou, Zhengtong Xu, Zhiyuan Zhang, Qiang Qiu, Yu She

arXiv:2505.18472v2cs.RO

TL;DR

Visuotactile policy learning lacks a reproducible, scalable benchmark for contact-rich and visually constrained manipulation. ManiFeel addresses this gap with a diverse task suite and modular evaluation across sensing, tactile representations, and policies, validated in simulation and real-world settings. Results show that tactile benefits and modality strengths depend strongly on task characteristics, while tactile representations and fusion architectures remain open challenges.

  • Problem

    Visuotactile policy learning lacks standardized, reproducible simulation infrastructure for rigorous comparative evaluation.

    Source: §I · p. 1

  • Method

    ManiFeel combines 13 manipulation task setups with modular evaluation of sensing modalities, tactile representations, and policy architectures, plus real-world validation.

    Source: §III · p. 3, A. Discussion and Insights · p. 13

  • Results

    Tactile sensing consistently improves performance in contact-rich and visually limited scenarios, with TacFF and TacRGB showing task-dependent strengths.

    Source: A. Discussion and Insights · p. 13

  • Takeaways & Limitations

    Effective visuotactile policy design requires matching tactile modalities to task interaction cues and developing more generalizable representations and adaptive fusion.

    Source: A. Discussion and Insights · p. 13, B. Limitations and Future Work · pp. 13–14

    Existing tactile representations do not generalize consistently across tasks, and comprehensive evaluation of all available methods is beyond the current scope.

    Source: B. Limitations and Future Work · p. 13

Abstract

from arXiv · show

Supervised visuomotor policies have shown strong performance in robotic manipulation but often struggle in tasks with limited visual inputs, such as operations in confined spaces and dimly lit environments, or tasks requiring precise perception of object properties and environmental interactions. In such cases, tactile feedback becomes essential for manipulation. While the rapid progress of supervised visuomotor policies has benefited greatly from high-quality, reproducible simulation benchmarks in visual imitation, the visuotactile domain still lacks a similarly comprehensive and reliable benchmark for large-scale and rigorous evaluation. To address this, we introduce ManiFeel, a reproducible and scalable simulation benchmark designed to systematically study supervised visuotactile policy learning. ManiFeel offers a diverse suite of contact-rich and visually challenging manipulation tasks, a modular evaluation pipeline spanning sensing modalities, tactile representations, and policy architectures, as well as real-world validation. Through extensive experiments, ManiFeel demonstrates how tactile sensing enhances policy performance across diverse manipulation scenarios, ranging from precise contact-driven operations to visually constrained settings. In addition, the results reveal task-dependent strengths of different tactile modalities and identify key design principles and open challenges for robust visuotactile policy learning. Real-world evaluations further confirm that ManiFeel provides a reliable and meaningful foundation for benchmarking and future visuotactile policy development. To foster reproducibility and future research, we will release our codebase, datasets, training logs, and pretrained checkpoints, aiming to accelerate progress toward generalizable visuotactile policy learning and manipulation.

I. INTRODUCTION

Visuotactile manipulation lacks the standardized, reproducible simulation infrastructure available for visual imitation learning. ManiFeel addresses this gap with a diverse benchmark, modular comparisons, empirical analysis, and real-world reproduction.

I. INTRODUCTION

  • Vision-only policies struggle when visual cues are absent, degraded, or insufficient for precise contact interactions, motivating complementary tactile feedback.Tactile sensing captures contact geometry, force distribution, and interaction dynamics that vision alone cannot perceive.Source: §I · p. 1
  • Visuotactile policy learning lacks comparable standardized benchmarks because high-fidelity tactile simulation is difficult, limiting controlled comparisons and systematic understanding.The gap also hinders exploration of robust policy principles and future directions.Source: §I · p. 1, A. Imitation Learning · p. 2
  • ManiFeel provides diverse contact-rich and visually challenging tasks with human demonstrations spanning insertion, assembly, exploration, and sorting.The suite is designed to evaluate how tactile information contributes to multimodal policy learning.Source: §I · p. 1
  • Its modular pipeline separates sensing modalities, tactile representations, and policy architectures for systematic combinations and comparisons.This design supports investigation of which model structures benefit visuotactile learning.Source: §I · pp. 1–2
  • Experiments vary modalities, tactile representations, and policies across simulation and real-world settings to identify task-dependent patterns and design principles.The benchmark also highlights challenges and promising directions for future research.Source: §I · p. 2
  • Real-world and simulation results correlate, supporting ManiFeel’s reliability and practical relevance as a reproducible benchmark.The platform is open-sourced with code, datasets, and pretrained checkpoints to enable fair, extensible comparisons.Source: §I · p. 2

B. Visuotactile Manipulation

Prior visuotactile manipulation methods are often task-specific, while tactile representation learning has been evaluated less systematically for policy learning. ManiFeel provides a standardized framework spanning diverse tasks and modular policy evaluation.

B. Visuotactile Manipulation

  • Existing visuotactile controllers commonly target specific tasks or object categories, limiting generalization across tasks and objects.Imitation learning offers a more flexible alternative by learning policies directly from demonstrations.Source: B. Visuotactile Manipulation · p. 2

C. Tactile Representation Learning

  • Tactile representation research includes CNN, masked autoencoding, VQGAN, transferable, sensor-invariant, and multimodal approaches.These methods aim to extract structured or generalizable information from tactile signals.Source: C. Tactile Representation Learning · pp. 2–3
  • Most tactile representation studies emphasize perception tasks such as classification and pose estimation, leaving policy-learning comparisons less explored.ManiFeel addresses this by standardizing scalable evaluation of tactile representations within policy learning.Source: C. Tactile Representation Learning · p. 3

III. MANIFEEL: VISUOTACTILE MANIPULATION LEARNING BENCHMARK

  • ManiFeel contains 13 task setups across insertion, screwing, and exploration, including 9 simulation and 4 real-world setups.The tasks vary in their reliance on visual and tactile modalities.Source: §III · p. 3
  • The benchmark’s modular pipeline enables systematic evaluation of sensing modalities, tactile representations, and policies across the task suite.This provides a unified basis for comparing visuotactile learning choices.Source: §III · p. 3

A. Task Suite and Dataset

ManiFeel organizes supervised visuotactile learning around insertion, screwing, and exploration tasks that vary in contact demands and visual availability. The suite includes simulation tasks and real-world reproductions for selected scenarios.

Task and Environment Design

  • The task suite spans insertion, screwing, and exploration, covering contact-rich assembly, rotational interactions, occluded search, reorientation, and dim-light sorting.These categories vary the relative need for visual and tactile information.Source: 1) Task and Environment Design · pp. 3–4
  • Insertion tasks require precise alignment and constrained motion for peg, USB, power-plug, and gear assembly scenarios.Policies learn coordinated motion and fine alignment from visuotactile demonstrations under pose and geometry variation.Source: 1) Task and Environment Design · p. 4
  • Screwing tasks test rotational and torque-sensitive interactions through nut–bolt assembly and bulb installation.Policies must regulate grasp force, rotation, threading, and tightening completion, including visually occluded interfaces.Source: 1) Task and Environment Design · p. 4
  • Peg reorientation requires contact-guided in-hand rotation before insertion when the socket interior is visually inaccessible.The task uses external contact with the socket rim for self-alignment.Source: 1) Task and Environment Design · p. 4
  • Object search requires tactile exploration to locate, grasp, and lift an unseen object inside a closed container.The policy detects contact events and regulates grasping without visual access.Source: 1) Task and Environment Design · p. 4
  • Ball sorting tests tactile identification of visually similar balls through their distinct surface textures under normal and dim lighting.The task is implemented in simulation and the real world to examine simulation fidelity.Source: 1) Task and Environment Design · p. 4

2) Observation modalities:

ManiFeel combines configurable visual observations with TacRGB and TacFF tactile channels, while demonstrations cover contact-rich insertion and exploration under occluded or degraded vision. Tactile feedback supports operator perception and fine-grained manipulation during data collection.

2) Observation modalities:

  • TacRGB captures elastomer deformation patterns related to surface texture and geometry, whereas TacFF encodes distributed normal and shear forces.The two channels provide complementary information about contact geometry and interaction dynamics.Source: 2) Observation modalities · pp. 4–5
  • Simulated tactile observations are compared with real GelSight measurements using tactile RGB and force-field signals across benchmark objects.The comparison assesses whether simulated feedback resembles physical sensor observations.Source: Figure 2 · p. 5, 2) Observation modalities · p. 5
  • The framework supports front, side, and wrist-mounted cameras alongside occluded or degraded visual conditions.Camera, lighting, and observation configurations vary across tasks.Source: 2) Observation modalities · p. 5

3) Teleoperated Demonstration Data Collection:

  • Teleoperators use real-time tactile observations to perceive subtle contact events during precise-contact and visually degraded tasks.Tactile feedback is especially important under occlusion and dim lighting.Source: 3) Teleoperated Demonstration Data… · p. 5
  • Insertion demonstrations illustrate tactile assistance for estimating in-hand pose and detecting misalignment invisible to external cameras.These cues support fine-grained behaviors and high-quality supervised demonstrations.Source: Figure 3 · p. 5, 3) Teleoperated Demonstration Data… · pp. 5–6
  • Exploration demonstrations cover peg reorientation, object search, and ball sorting with severely degraded or occluded visual input.The agent view visualizes occlusion but is excluded from the demonstration dataset.Source: Figure 4 · p. 6

B. Modular Pipeline Design

ManiFeel uses a modular benchmark pipeline that independently compares sensing configurations, tactile representations, and policy architectures across visuotactile manipulation tasks. Dedicated modality encoders fuse visual, tactile, and proprioceptive observations before policy networks generate action chunks.

Pipeline Design

  • ManiFeel enables systematic evaluation of visuotactile policies across sensing modalities, tactile representations, and policy architectures, supporting flexible combinations and fair comparisons.Its modularity also accommodates future sensing modalities, tactile representation methods, and policy architectures.Source: B. Modular Pipeline Design · p. 6

Choice of Modalities

  • Three sensing configurations compare vision-only, vision with TacRGB, and vision with TacFF while avoiding confounding interactions between tactile modalities.A tactile-only baseline is excluded because benchmark tasks generally require vision for global localization and pre-contact alignment.Source: B. Modular Pipeline Design · p. 6

Observation Fusion

  • Visual, tactile, and proprioceptive observations are processed by separate encoders, whose latent features are concatenated and passed to a policy network to generate action chunks.The encoder receives current and previous observations from each modality.Source: B. Modular Pipeline Design · pp. 6–7, Figure 5 · p. 7

Tactile Representations

  • The benchmark compares four tactile representations: scratch-trained ResNet18, pretrained T3, lightweight UniT, and generalizable AnyTouch.These comparisons assess whether pretrained or carefully designed tactile encoders provide advantages for policy learning.Source: B. Modular Pipeline Design · p. 7

Policies

  • ManiFeel evaluates Diffusion Policy, Equivariant Diffusion Policy, and Flow Matching to study how policy architectures interact with tactile inputs.The policies differ in generative modeling approaches and model architectures.Source: B. Modular Pipeline Design · p. 7

IV. EXPERIMENTS AND RESULTS

The experiments benchmark sensing configurations across nine simulation tasks, showing that tactile feedback improves both contact-rich manipulation and visually constrained exploration. The gains are task-dependent: force-based TacFF favors alignment and force regulation, whereas TacRGB favors texture- and geometry-driven recognition.

A. Modality Benchmarking

  • TacFF increases peg insertion success by 26 percentage points and bulb installation by about 5 percentage points, while TacRGB improves object search by approximately 17 and dim-light ball sorting by around 15 percentage points versus vision only.These success rates average approximately 1500 evaluations across seeds, initializations, and training epochs.Source: A. Modality Benchmarking · p. 8, Table II · p. 8
  • TacFF best supports insertion and screwing because distributed normal and shear forces provide physically grounded cues for stable contact, fine alignment, and force regulation.The modality is therefore matched to contact-driven manipulation demands.Source: A. Modality Benchmarking · p. 8
  • TacRGB best supports object search and ball sorting because high-resolution tactile images capture surface texture and shape information when visual cues are occluded or degraded.In dim-light ball sorting, success rises from 57% with vision only to 72% with TacRGB, versus 60% with TacFF.Source: A. Modality Benchmarking · p. 8
  • Adding tactile data can reduce performance when the modality is not task-relevant: gear assembly falls from 63% with vision only to 57% with TacRGB.The authors attribute this to noise, overfitting, or attention to semantically weak deformation patterns.Source: A. Modality Benchmarking · p. 8
  • Rollouts show TacFF correcting subtle contact deviations during partially occluded plug insertion, while TacRGB distinguishes visually similar balls under dim lighting.The examples connect each modality's empirical advantage to its task-relevant sensing signal.Source: A. Modality Benchmarking · p. 9, Figure 7 · p. 9
  • Overall, effective visuotactile learning requires selecting structured, task-relevant tactile representations rather than assuming that more sensory input is always beneficial.This conclusion follows across contact-rich and visually ambiguous conditions.Source: A. Modality Benchmarking · pp. 8–9

B. Policy Benchmarking

Policy benchmarking compares DP, EquiDP, and FM across representative manipulation tasks and sensing configurations. Tactile benefits remain consistent across architectures but depend on modality-task alignment and policy design.

B. Policy Benchmarking

  • TacFF generally improves contact-rich tasks, while TacRGB yields the largest gains in exploration: EquiDP improves by 23 and 25 percentage points on object search and ball sorting, and FM improves by 28 percentage points on both tasks.These comparisons support the complementary roles of force-based and high-resolution image-based tactile feedback.Source: B. Policy Benchmarking · p. 9, Table III · p. 10
  • In gear assembly, TacFF increases success by about 3 percentage points for DP and 4 percentage points for EquiDP relative to vision-only baselines.The gains reflect the value of force feedback for precise alignment and contact regulation.Source: B. Policy Benchmarking · p. 9
  • Across architectures, FM performs worse than DP and EquiDP in most tasks, while EquiDP is strongest in insertion and screwing and DP can be more robust when visual cues are unreliable.EquiDP's advantage is linked to spatially equivariant visual processing, whereas exploration relies more heavily on tactile-driven fusion.Source: B. Policy Benchmarking · p. 10
  • TacRGB can degrade contact-rich performance across architectures when its high-dimensional deformation images are not task-relevant, although architecture choice changes sensitivity rather than removing the mismatch.The authors report degradation in gear assembly across all architectures and pronounced sensitivity for DP in bulb installation.Source: B. Policy Benchmarking · p. 10
  • The results establish that tactile sensing improves performance across policy architectures, but the magnitude and direction of improvement depend jointly on tactile modality and policy design.Modality selection should therefore precede fine-grained architecture optimization when the task's sensing requirements are known.Source: B. Policy Benchmarking · p. 10

C. Tactile Representation Benchmarking

Pretrained tactile encoders improve vision–TacRGB policies, but their strengths vary by task. Real-world experiments reproduce simulation trends, with force fields benefiting force-sensitive tasks and tactile images aiding visually ambiguous sorting.

C. Tactile Representation Benchmarking

  • UniT and T3 consistently outperform a scratch-trained ResNet18, especially on object search and ball sorting requiring contact or texture reasoning.UniT provides the most balanced performance across gear assembly, object search, and ball sorting, while AnyTouch is strong for texture reasoning but weaker for force-sensitive control.Source: C. Tactile Representation Benchmarking · p. 11
  • Pretrained tactile representations improve policy learning, but inconsistent cross-task performance leaves unified tactile generalization unresolved.Performance does not transfer consistently across tasks and modalities, motivating more robust representations for spatial contact and texture cues.Source: C. Tactile Representation Benchmarking · p. 11

D. Real-world Evaluation

  • TacFF improves real gear-assembly success by 14 percentage points over vision only, from 33% to 47%, mirroring the simulation trend.The result supports force-field feedback for detecting gear misalignment during contact-rich insertion.Source: D. Real-world Evaluation · p. 12
  • TacFF achieves the highest bulb-installation success, reaching 73% in real experiments and 81% in simulation.The result is consistent with force-based feedback helping regulate tightening and contact.Source: D. Real-world Evaluation · p. 12
  • TacRGB raises real ball-sorting success under normal lighting from 33% to 80%, matching the simulation improvement from 70% to 85%.Under dim lighting, localization remains difficult before contact, but TacRGB supports stable classification and transport after contact.Source: D. Real-world Evaluation · pp. 12–13
  • Across real and simulated settings, relative performance trends remain consistent, supporting ManiFeel as a meaningful benchmark despite nonidentical quantitative results.The benchmark reflects representative manipulation behaviors rather than exact numerical equivalence between domains.Source: D. Real-world Evaluation · p. 13

V. DISCUSSION AND LIMITATIONS

ManiFeel shows that tactile sensing improves manipulation when contact dominates or vision is ambiguous, while the best modality depends on task cues. Simulation trends generally transfer to reality, but lighting, sensor, and contact-model differences create quantitative gaps.

A. Discussion and Insights

  • Tactile sensing consistently improves performance in contact-rich tasks and settings with limited or ambiguous visual input across simulation and real experiments.The tactile signal supplies complementary feedback for robust control where vision is uncertain.Source: A. Discussion and Insights · p. 13
  • TacFF is strongest for geometric fitting and force-sensitive interactions, whereas TacRGB is better for object pose and surface-texture recognition under limited vision.The contribution of each modality depends on alignment between its representation and the task’s dominant physical interaction cues.Source: A. Discussion and Insights · p. 13
  • Real-world results closely mirror simulation trends, but severe lighting degradation, tactile-signal differences, elastomer deformation, and frictional dynamics produce quantitative discrepancies.These gaps motivate continued work on sim-to-real visuotactile sensing.Source: A. Discussion and Insights · p. 13
  • Modality selection, action-head design, visual-encoder choice, and tactile-representation quality all materially affect visuotactile policy performance.DP outperforms FM under matched encoder and fusion settings, while UniT and T3 outperform task-specific CNNs; equivariant visual encoders help some contact-rich tasks but can underperform when vision is unreliable.Source: A. Discussion and Insights · p. 13

B. Limitations and Future Work

ManiFeel provides an extensible benchmark but remains bounded by incomplete tactile generalization, limited modality combinations, representative rather than exhaustive task coverage, and sim-to-real discrepancies. These boundaries define clear directions for broader representations, adaptive fusion, and improved sensor modeling.

B. Limitations and Future Work

  • Existing tactile representations do not generalize consistently across tasks, and comprehensive evaluation of all available methods remains outside the current scope.The benchmark is modular and can be extended with additional encoders, representations, and policy architectures.Source: B. Limitations and Future Work · p. 13
  • ManiFeel evaluates vision paired with one tactile modality at a time and uses concatenation-based fusion, leaving advanced multimodal and task-aware fusion unexplored.Combining TacRGB and TacFF may not outperform the best pairwise combination and would complicate attribution.Source: B. Limitations and Future Work · pp. 13–14
  • Representative tasks preserve key category-level contact dynamics but do not provide full task coverage.The selected tasks are intended to support feasible, systematic comparisons and reasonably generalize to similar manipulations.Source: B. Limitations and Future Work · p. 14
  • Simulation and real experiments largely agree in trend but differ in absolute success rates because of tactile-signal fidelity, camera sensitivity, unmodeled contact dynamics, and setup differences.The real-world experiments validate overall consistency rather than direct policy transfer, especially for ball sorting.Source: B. Limitations and Future Work · p. 14

VI. CONCLUSION

  • ManiFeel establishes a reproducible and extensible foundation for studying sensing modalities, tactile representations, policy architectures, and multimodal fusion.Its modular design supports future work toward more robust and generalizable visuotactile manipulation.Source: §VI · p. 14

APPENDIX A ADDITIONAL DETAILS ON TASK ENVIRONMENTS

The benchmark spans randomized insertion, screwing, reorientation, search, and sorting environments, with matched real-world gear, bulb, and ball-sorting tasks. Its dataset records synchronized multimodal demonstrations across simulation and reality for flexible policy evaluation.

APPENDIX A ADDITIONAL DETAILS ON TASK ENVIRONMENTS

  • Simulation task configurations are built on IsaacGym, and code and configuration files are intended for open-source release to support reproducibility.The dataset summary defines camera, tactile, proprioceptive, action, and demonstration fields used to characterize each task.Source: Appendix A · p. 14, Table V · p. 15
  • Simulation environments randomize object or socket positions and relevant orientations to test alignment, insertion, screwing, reorientation, search, and sorting behaviors.Object search is fully occluded and tactile-only, while dim-light ball sorting tests degraded visual perception.Source: Appendix A · p. 14
  • Real gear assembly introduces in-plane base rotation, while bulb installation randomizes initial screw depth and therefore the required tightening turns.These variations create uncertainty in alignment and tightening across rollouts.Source: Appendix A · pp. 14–15

APPENDIX B DATASET DETAILS

  • The benchmark collects expert demonstrations for 9 simulation and 4 real-world tasks, recording task-specific action dimensions and constrained rotations.A 7-dimensional action space includes gripper control, whereas real-world experiments use an action dimension of 5 with fixed roll and pitch.Source: Appendix B · p. 15
  • Simulation data includes three RGB views and two tactile sensors, while real data uses a front-facing RealSense camera and one right-finger GelSight R1.5 sensor.Observations are synchronized at 10 Hz and include visual, tactile, and proprioceptive feedback.Source: Appendix B · p. 15

APPENDIX C PIPELINE DETAILS

The pipeline combines separate visual and tactile encoders with multiple tactile representations and policy architectures, evaluated across standardized simulation and real-world setups.

A. Visual and Tactile Encoders

  • Separate ResNet18 encoders process each camera view and extract tactile embeddings from images or force fields without pretrained tactile representations.Spatial softmax preserves spatial information, while Group Normalization supports stable training.Source: A. Visual and Tactile Encoders · p. 15

B. Tactile Representations

  • Tactile representations span customized UniT models, pretrained T3 and AnyTouch encoders, and direct ResNet18 features, with AnyTouch using 305 M parameters.UniT is trained separately for simulation and real-world tactile domains, while T3 and AnyTouch checkpoints are reused across both settings.Source: B. Tactile Representations · pp. 15–16

C. Policies

  • The evaluated policies include CNN-based Diffusion Policy, SO(2)-equivariant Diffusion Policy, and one-step Flow Matching for action generation.Diffusion Policy uses a 1D UNet, EquiDP uses relative control, and Flow Matching avoids iterative denoising.Source: C. Policies · p. 16

A. Experimental Setup

  • Experiments vary cameras, tactile sensors, action dimensions, demonstrations, lighting, and occlusion across simulation and real-world tasks, using only the right-finger tactile sensor.Gear assembly performance saturates around 80 demonstrations, which is adopted for comparative experiments; bulb installation uses 20 demonstrations and exceeds 70% simulation success.Source: A. Experimental Setup · pp. 16–17, Table VI · p. 16, Figure 12 · p. 16

B. Training Details

Training uses standardized image and tactile preprocessing with diffusion-specific inference settings, while runtime and viewpoint experiments assess computational cost and camera-dependent performance.

B. Training Details

  • Tactile inputs are resized according to encoder requirements, visual inputs remain 256×256, and DDIM replaces DDPM for faster real-world Diffusion Policy inference.T3 and AnyTouch use 224×224 tactile images, while ResNet18 and UniT use 256×256 inputs.Source: B. Training Details · p. 17

C. Runtime Analysis of Tactile Representations

D. Experiment on Different Camera Viewpoints

  • Wrist-mounted inputs yield higher object-search success than front-facing inputs for both policy types, while visuotactile policies outperform vision-only policies under both viewpoints.The viewpoints represent dim wrist views and occluded front views, with final success rates averaged over the last ten epochs and three seeds.Source: D. Experiment on Different Camera… · p. 17, Figure 13 · p. 17
Loading 2505.18472v2…