PaperlayerView exact revision on arXiv

Source-linked AI summary

PneuTac: Tactile Manipulation with Soft Pneumatic Robots via Unified MPM-Gaussian Splatting Simulation

Shaohong Zhong, Marco Pontin, Joe Watson, Perla Maiolino, Ingmar Posner

arXiv:2609.38418v1cs.RO

TL;DR

Learning tactile manipulation with compliant robots is challenging because demonstrations are costly and existing simulators typically treat soft robots and tactile sensors separately. PneuTac combines MPM dynamics, 3DGS rendering, and vision-based calibration with surrogate simulation and tactile-guided demonstration augmentation. Its augmented demonstrations improve real-world policy success over training with the same amount of real data, while direct policy comparison with another simulator is unavailable.

  • Problem

    Learning tactile manipulation with compliant robots is limited by costly real demonstrations and the lack of simulators that jointly model compliant actuation and tactile sensing.

    Source: §I · p. 1

  • Method

    PneuTac combines MPM models of soft-robot and tactile-gel dynamics with 3DGS rendering, vision-based calibration, surrogate networks, and tactile-guided demonstration augmentation.

    Source: §I · p. 1, A. MPM-3DGS Simulation · p. 2

  • Results

    PneuTac-augmented demonstrations substantially increase real-world policy success rates over training with the same amount of real-world demonstrations.

    Source: C. Contact-Rich Compliant Manipulation · p. 7

  • Takeaways & Limitations

    The framework demonstrates practical utility for tactile manipulation on compliant hardware through real-world contact-rich manipulation tasks.

    Source: §VI · p. 8

    A direct policy baseline comparison with another simulator is unavailable because few existing frameworks natively model the paired devices.

    Source: C. Contact-Rich Compliant Manipulation · p. 7

Abstract

from arXiv · show

Soft robots and tactile sensors have demonstrated great potential in delicate manipulation tasks. Soft pneumatic robots enable safe contact through compliance, and vision-based tactile sensors offer high-resolution touch perception. However, learning tactile manipulation with compliant robots has been challenging, bottlenecked by the lack of efficient simulation. Existing simulators typically model them in isolation, and exhibit large calibration gaps that are difficult to overcome efficiently. We present PneuTac, a unified framework for tactile-feedback manipulation with soft pneumatic robots. We leverage the material point method (MPM) for modelling the dynamics of the soft robot and the deformable tactile membrane, and 3D Gaussian splatting (3DGS) for rendering. Real-to-sim modelling is done with a simple vision-based method, to then train action and perception networks for efficient simulation with surrogate models. We use the framework to drive a tactile-guided pipeline to collect demonstrations in simulation. Through experiments on a custom-designed pneumatic soft finger with a tactile sensing tip, together with additional cross-device evaluations, we show that PneuTac is capable of accurately modelling soft robots with tactile sensors, and that policies trained with simulation-augmented demonstrations outperform baselines trained on the same real data on three real-world contact-rich compliant manipulation tasks, making it a practical framework for tactile manipulation on compliant hardware.

I. INTRODUCTION

Learning tactile manipulation with compliant robots is hindered by costly real demonstrations and simulators that model soft robots and tactile sensors separately. PneuTac combines their simulation and uses it to support tactile-guided policy learning.

I. INTRODUCTION

  • Real demonstrations for contact-rich tasks are slow and expensive to collect, while existing simulators generally model compliant robots and tactile sensors separately.Repeated calibration may also be needed as materials or devices degrade.Source: §I · p. 1
  • PneuTac couples MPM dynamics with 3DGS appearance modelling and uses tactile-guided simulation demonstrations to train manipulation policies for soft pneumatic robots.The framework is also evaluated across distinct robot morphologies and tactile sensor instances.Source: §I · p. 1

A. Vision-based tactile simulation

  • Prior tactile simulators typically couple sensors to rigid robot and object physics, whereas PneuTac supports tactile simulation on a soft robot using a single indentation image for rendering calibration.The overview describes the framework as supporting soft-robot deformation and tactile-contact rendering from CAD models.Source: A. Vision-based tactile simulation · p. 2, Figure 1 · p. 2

B. Soft robot simulation

PneuTac extends simulation for soft robots and physics-integrated Gaussian rendering to actuated robots with tactile sensing. Its methodology combines MPM for deformable dynamics with 3DGS rendering and a pipeline for calibration, surrogate modelling, and demonstration collection.

B. Soft robot simulation

  • Soft-robot simulators have rarely supported tactile sensing for manipulation policy learning, and PneuTac extends this line toward tactile-based manipulation.Prior approaches required substantial calibration effort to overcome the sim-to-real gap.Source: B. Soft robot simulation · p. 2

C. Physics-integrated Gaussian rendering

  • Unlike prior physics-integrated Gaussian work on deformable objects, PneuTac applies MPM-3DGS to actuated robots with tactile sensing for sim-to-real policy learning.The related-work passage characterizes that combination as not achieved by prior works.Source: C. Physics-integrated Gaussian rendering · p. 2

III. METHODOLOGY

  • The framework combines an MPM-3DGS simulator, vision-based calibration, action and perception surrogate networks, and tactile-guided demonstration collection from a small set of real demonstrations.The pipeline overview identifies calibration, surrogate simulation, and simulated demonstration generation as its stages.Source: §III · p. 2, Figure 2 · p. 3

A. MPM-3DGS Simulation

  • MPM models the soft robot and deformable tactile gel, while coupled 3D Gaussians render deformation and tactile images, including contact-dependent lighting effects.The MPM representation tracks particles and resolves dynamics on a background grid; the contact photometric model depends on gel slope and indentation depth.Source: A. MPM-3DGS Simulation · pp. 2–3

B. Real-to-Sim Modelling

Real-to-sim modelling calibrates the soft robot from equilibrium images and calibrates tactile contact rendering from a rest image and an indentation image. The contact model adjusts Gaussian color according to indentation slope and depth.

B. Real-to-Sim Modelling

  • Robot calibration identifies MPM material and actuation parameters by matching simulated and observed equilibrium deformation under fixed pneumatic pressures.The actuation parameters map input pressure to joint torque, and parameter identification uses a sampling-based image-loss method.Source: B. Real-to-Sim Modelling · p. 3
  • Tactile calibration fixes the rest-image 3DGS model, then uses a single registered indentation image and matching MPM deformation to estimate contact-rendering parameters.The indentation image is paired with a rest background image.Source: B. Real-to-Sim Modelling · p. 3
  • The contact photometric model adds Gaussian color increments linear in gel surface slope and normalized indentation depth.Its coefficients are estimated by minimizing the difference between the rendered and observed indentation images.Source: B. Real-to-Sim Modelling · p. 3

C. Surrogate Models with Action and Perception Networks

PneuTac combines MPM-3DGS modeling with learned surrogates and tactile-guided simulation to support calibration and demonstration collection across soft-robot devices.

C. Surrogate Models with Action and Perception Networks

  • Two learned surrogates replace full MPM runs during efficient simulation: a residual-torque model for soft-robot actuation and a probe-grid-to-MPM mapping for tactile rendering.The full MPM-3DGS models remain available for calibration.Source: C. Surrogate Models with Action and… · p. 3

D. Tactile-guided Demonstration Collection

  • For contact-phase demonstration collection, a tactile-guided CEM model-predictive controller uses real trajectories as warm starts and encourages agreement with demonstrated contact-area trajectories.The simulated demonstrations retain the real-world observation and action spaces.Source: D. Tactile-guided Demonstration… · pp. 3–4

IV. EXPERIMENTS

  • Evaluation uses a custom one-chamber soft finger with a Digit sensor, plus a structurally different three-chamber soft arm and an unmodified Digit sensor for cross-device evaluation.The soft arm uses a six-joint formulation and a side view to capture out-of-plane bending.Source: §IV · p. 4, 1) Soft robot · pp. 4–5

1) Soft robot:

  • The simulator calibrates soft-finger parameters against real images at preset pressures, using appearance and silhouette agreement and held-out pressure levels for evaluation.The model also uses per-joint gain parameters to distribute bending along the spine and a residual rest pressure to capture un-actuated bend.Source: 1) Soft robot · p. 4

2) Tactile rendering:

Tactile calibration fits a contact photometric model from a real indentation and evaluates rendered tactile images against held-out sensor images and established models.

2) Tactile rendering:

  • The coupled 3DGS model is trained from one background image of the sensor, using known Digit camera parameters and known gel material parameters.The gel MPM model is initialized by regular-grid particle sampling within a box matching the gel dimensions.Source: 2) Tactile rendering · p. 5
  • The contact photometric model fits slope-related and indentation-depth coefficients from one real indentation image paired with simulated MPM deformation, then refines them through rendering optimization.The same procedure is applied to the finger-mounted and unmodified Digit sensors.Source: 2) Tactile rendering · p. 5
  • Tactile rendering is evaluated on 20 held-out real indentation images using RMSE, PSNR, and NCC, against reimplementations of Taxim, Tacchi, and DOT-Sim.The evaluation uses the same training and evaluation datasets for the compared models.Source: 2) Tactile rendering · p. 5

B. Action and Perception Networks

The paper trains action and perception networks on simulated data to approximate MPM actuation and tactile deformation efficiently.

B. Action and Perception Networks

  • The action network predicts residual joint torque from spine position, velocity, and pressure, which is added to the MPM actuation torque for the skeleton model.Training uses 8,000 simulated state–torque pairs and inverse-dynamics targets.Source: B. Action and Perception Networks · p. 5
  • The perception network learns to map probe indentation-depth observations to the corresponding MPM deformation field from paired contact simulations.The training set contains 100 samples with varied indentation shapes, depths, rotations, and positions.Source: B. Action and Perception Networks · p. 5
  • The perception model is a three-level convolutional encoder-decoder with ReLU activations and skip connections, trained by minimizing contact-region MSE.The action network is a fully connected MLP with two hidden layers of width 128.Source: B. Action and Perception Networks · p. 5

C. Contact-Rich Compliant Manipulation

The manipulation pipeline collects real demonstrations for three delicate tasks and bootstraps additional demonstrations in simulation.

C. Contact-Rich Compliant Manipulation

1) Demonstration collection in simulation:

The section describes simulation-based demonstration collection and behavior-cloning policy training, using tactile feedback to guide trajectories and combining simulated and real demonstrations.

1) Demonstration collection in simulation:

  • The GPU-parallel CEM-MPC planner tracks demonstrated tactile-area and success trajectories while penalizing jerky actions, with domain randomization across four factors.The tactile-area signal is thresholded contact pixels divided by total pixels; randomization perturbs friction, object position, peak pressure, and pressure-to-bend gain.Source: 1) Demonstration collection in… · p. 5

2) Policy training:

  • Policy training uses behavior cloning to predict Cartesian displacement and finger pressure from tactile images and scalar observations, with simulated demonstrations followed by real-data fine-tuning.The observation includes contact area, pressure, joint states, end-effector position, and the previous action.Source: 2) Policy training · pp. 5–6

V. RESULTS AND DISCUSSION

PneuTac is evaluated on real-to-sim model quality, surrogate efficiency, and tactile-guided demonstrations for compliant manipulation. The supplied results show accurate soft-robot motion and rendering, including at a viewpoint opposite calibration.

A. Real-to-Sim Modelling

  • The model uses about 4k particles for the soft robot and 60k for the gel, whose higher resolution is needed to capture deformation under contact.The particle models are sampled from available CAD models.Source: A. Real-to-Sim Modelling · p. 6

1) Soft robot finger:

  • Vision-based calibration identifies physically realistic parameters, with soft-material elasticity in the printer’s expected regime and p0 accounting for gravity-induced rest curl.The calibrated model uses MPM with a rigid spine to simulate soft-finger behavior.Source: 1) Soft robot finger · p. 6
  • PneuTac reproduces bending for the soft finger and arm and matches the real finger appearance from the back, opposite the calibration camera.The paper reports realistic front- and back-view rendering relative to baseline methods.Source: 1) Soft robot finger · p. 6, Figure 3 · p. 6, Table I · p. 6
  • 4.2 ± 0.4 mm finger tip error (<4% of finger length) and 4.9 ± 1.0 mm arm tip error (4.2% of arm length) occur at unseen pressure points.Measurements use marker-based real-world tip positions; the errors are commensurate with prior approaches on similar platforms.Source: 1) Soft robot finger · p. 6

2) Tactile rendering:

PneuTac produces tactile observations that outperform alternative rendering methods across evaluated image metrics and generalize to a second Digit sensor. Action and perception surrogates approximate the MPM models while enabling faster simulation.

2) Tactile rendering:

  • PneuTac outperforms baseline models across all evaluated tactile-image metrics on the same training dataset.Table II compares tactile-rendering methods; the reported results concern the small-calibration-data regime.Source: 2) Tactile rendering · p. 6, Table II · p. 7
  • On a second Digit sensor, evaluation images score RMSE 8.12 ± 0.22, PSNR 29.97 ± 0.22 dB, and NCC 0.976 ± 0.001, on par with the original sensor.Reusing the original calibration directly gives RMSE 22.96, reflecting device differences.Source: 2) Tactile rendering · p. 7
  • The tactile model captures the contact-driven lighting shift from redder to greener image regions across the two Digit sensors.Figure 4 columns 1–3 show the sensor attached to the soft finger; columns 4–5 show an unmodified Digit sensor.Source: 2) Tactile rendering · p. 7, Figure 4 · p. 7

B. Action and Perception Networks

  • The residual-torque skeleton model reaches 0.051 ± 0.006° mean joint-position error and 2.76 ± 0.55°/s velocity error, with a 5x speedup over full MPM.The action network takes about 10 ms versus about 50 ms for full MPM.Source: B. Action and Perception Networks · p. 7
  • The perception network maps probe indentation to MPM with 0.077 ± 0.005 mm mean absolute error per contact point; the probe model runs 3000x faster than full MPM.The mean indentation depth is 1.4 mm; simulation takes about 50 ms versus about 150 s for full MPM.Source: B. Action and Perception Networks · p. 7

C. Contact-Rich Compliant Manipulation

PneuTac-augmented demonstrations improve real-world tactile manipulation policies across contact-rich tasks, and the conclusion presents the unified framework as a practical engine for compliant hardware.

C. Contact-Rich Compliant Manipulation

  • Evaluation covers switch flipping, egg-carton opening, and card pulling, with 10 trials per seed across three seeds.Each task requires pressurising the pneumatic chamber and controlling the robot to contact the object.Source: C. Contact-Rich Compliant Manipulation · p. 7, Figure 5 · p. 8
  • PneuTac simulation demonstrations substantially increase real-world success over same-sized real-only training, particularly on egg-carton opening and card pulling.On the switch task, doubling the real demonstration budget matches Sim+Real.Source: C. Contact-Rich Compliant Manipulation · p. 7
  • Naive augmentation beats the real-only baseline but trails Sim+Real, while tactile- and trajectory-guided demonstrations provide useful policy pre-training information.On switch flipping, where 10 real demonstrations are already informative, doubling the real budget matches the simulation-augmented pipeline.Source: C. Contact-Rich Compliant Manipulation · p. 7
  • Tactile feedback substantially improves switch and egg-carton success, while card-pulling changes little; PPO matches BC on egg carton, trails on switch, and fails on card.The card-pulling pattern is consistent with that task being position- rather than contact-dominated; PPO's card failure is consistent with reduced simulator fidelity for card-on-card contact.Source: C. Contact-Rich Compliant Manipulation · p. 7
  • A direct policy comparison against another simulator is unavailable because few frameworks natively model the paired devices; PneuTac is among the first joint policy-learning frameworks.The paper describes this as opening the way to tactile-based compliant manipulation with soft robots.Source: C. Contact-Rich Compliant Manipulation · pp. 7–8

VI. CONCLUSION

  • The conclusion presents PneuTac as a unified MPM-3DGS framework for tactile-feedback manipulation, with efficient real-to-sim modelling, surrogate simulation, and demonstration-augmented policy learning.It reports validation across different soft robot and sensor devices and greater realism than prior baselines.Source: §VI · p. 8

Related papers

Loading 2609.38418v1…