Source-linked AI summary
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, Sergey Levine
TL;DR
The paper addresses tactile control for manipulation when objects are occluded and desired behavior is difficult to specify from touch. It learns deep predictive models from unsupervised interaction and uses tactile goal images to control a robot, demonstrating touch-only manipulation across three tasks.
Problem
Tactile manipulation lacks general-purpose control methods that use raw touch effectively, and desired behaviors are difficult to specify directly in tactile observations.
Method
Deep tactile MPC learns forward predictive models from raw GelSight observations and unsupervised exploratory interactions, then plans toward user-provided goal tactile images.
Results
The robot performed ball repositioning, joystick manipulation, and die reorientation using only touch sensing, with the method substantially outperforming a hand-designed baseline on ball repositioning metrics.
Takeaways & Limitations
Tactile predictive modeling can enable robotic fingers to reposition objects toward user-specified configurations despite visual occlusion.
Takeaways & Limitations
The method focuses on short-horizon control achievable within tens of time steps, limiting tasks requiring multiple objects or repeated complex finger gaits.
Abstract
from arXiv · showhide
Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading. Videos, visualizations and the code are available here: https://sites.google.com/view/deeptactilempc
I. INTRODUCTION
Tactile sensing is crucial for dexterous manipulation because contact forces and object interactions are often hidden from vision, yet robotic touch control remains difficult. The paper combines high-resolution sensing with learned predictive control to specify and achieve manipulation goals directly in tactile space.
- Motivation: Touch provides immediate contact information that vision often misses because the actuators occlude the interaction site.Tactile feedback can reveal friction, compliance, mass, and other physical properties relevant to manipulation.
- Challenges: Robotic tactile control remains challenging because sensors have often been sparse, contact physics are difficult to model, and tactile goals are hard to specify.
- Approach: The paper combines high-resolution GelSight sensing with deep models that can handle high-dimensional tactile observations and select actions.
- Goal specification: The method plans toward goals specified directly as raw tactile observations, which can be more informative and natural than sparse force-based goals.
- Contributions: The authors learn model-based control policies from unsupervised exploratory interactions without rewards and evaluate them on ball, joystick, and die manipulation.
- Results: The method outperforms hand-designed baselines on all three manipulation tasks, supporting touch-based control under visual occlusion.
II. RELATED WORK
Prior touch-control methods used hand-designed laws, extracted features, or low-dimensional sensing, while this work uses raw high-resolution GelSight observations with predictive control. The paper positions tactile video prediction as a new route to learned object repositioning.
- Prior methods: The approach differs from prior touch-control methods by learning general-purpose predictive models directly from high-resolution raw tactile observations rather than fixed laws or features.
- Tactile sensing: Compared with prior BioTac work using low-dimensional readings, this method operates on RGB GelSight images downsampled to 48x64 pixels for freer object repositioning.
- Predictive control: Video prediction with model-predictive control had previously been used for RGB-based non-prehensile repositioning, but not, to the authors’ knowledge, with touch sensing.
- Tactile sensing: GelSight provides detailed contact-surface images, though its resolution comes with a latency trade-off that the authors accept for precise repositioning.
- Novelty: The paper states that GelSight-style sensors had not previously been used for object repositioning with learned predictive models.
III. TASKS AND HARDWARE SETUP
The study evaluates tactile model-predictive control on ball repositioning, joystick deflection, and 20-sided die rolling using a high-resolution GelSight sensor and autonomous interaction data. The controller predicts tactile outcomes for candidate actions and executes the first action from the best sequence.
- Tasks: The three tasks are ball repositioning, analog-stick deflection, and rolling a 20-sided die to a specified upward face.
- Ball repositioning: Ball repositioning requires balancing contact forces so an underactuated ball reaches a target without slipping from the finger.
- Deep tactile MPC: The controller samples action sequences, predicts their tactile outcomes, evaluates distance to the goal observation, and applies the first action from the lowest-cost sequence.
- Analog-stick deflection: Joystick control requires lifting, repositioning, and reestablishing contact because sliding along the stick causes undesirable deflections.
- Die rolling: Die rolling is difficult because of slippage and unwanted rolling, and success is measured by the fraction of trials ending with the desired face upward.
- Data collection: Training data came from autonomous random movements, comprising 7400 ball trajectories, about 3000 joystick trajectories, and 4500 die trajectories.
IV. DEEP TACTILE MODEL-PREDICTIVE CONTROL
The method learns a deep predictive model of future GelSight observations and uses tactile MPC to plan actions toward user-provided tactile goals. Predictions are conditioned on the current tactile image and candidate action sequence, then replanned online to compensate for model inaccuracies.
- Deep predictive model: The predictive model maps the current tactile observation and candidate actions to future GelSight observations, accurately predicting contact patterns 13 time-steps ahead.The model is a deterministic deep recurrent convolutional network trained on tactile observations.
- Goal specification: At test time, a goal tactile image specifies the desired object configuration directly in the tactile observation space.The goal image is denoted Ig and supports tactile servoing without requiring an explicit object pose.
- Tactile MPC control: Tactile MPC optimizes action sequences whose predicted observations minimize pixel-space distance to the goal image.The controller uses mean squared error as the cost and cross-entropy-method sampling for optimization.
- Tactile MPC control: The controller executes only the first action of the best plan and replans at every real-world timestep to address predictive-model inaccuracies.The prediction horizon is 15–18 steps, with actions repeated three times and plans containing five or six actions.
- Tactile MPC control: The model predicts tactile outcomes for action sequences, while the rollout visualization compares the lowest-cost predicted sequence with the goal image and actual robot trajectory.The ball-bearing rollout displays every second predicted frame and records both tactile and side images during execution.
V. EXPERIMENTAL RESULTS
The experiments evaluate deep tactile MPC on three real-world manipulation tasks: moving a ball bearing, positioning an analog joystick, and rolling a die.
- Experimental evaluation: The evaluation covers three real-world tactile manipulation tasks: ball-bearing repositioning, analog joystick positioning, and die rolling.These tasks assess the method across distinct object-manipulation settings.
- Experimental evaluation: The die task tests whether the robot can roll an object to a specified face, while the ball and joystick tasks test tactile repositioning.The paper presents these tasks as the experimental validation of deep tactile MPC.
A. Evaluation Metrics
Evaluation uses tactile-image error, manually annotated pressure-centroid distance, and die-face success to measure different aspects of control. Because tactile goals do not reveal ground-truth object pose, image-space MSE alone is insufficient as a positional measure.
- Evaluation metrics: Tactile-image goals are general because users can specify desired configurations by manually positioning the sensor, but this makes ground-truth pose unavailable.The evaluation therefore compares tactile outcomes rather than directly measuring known object poses.
- Evaluation metrics: The evaluation measures final tactile-image MSE, manually annotated pressure-centroid distance, and the fraction of die trials reaching the desired face.The metrics cover image similarity, object-position error, and task-level success.
- Evaluation metrics: The analog-stick example shows a predicted action sequence whose final tactile image closely resembles the desired indentation after the pressure center lifts and returns.The figure displays every second prediction step for the selected optimal sequence.
- Evaluation metrics: MSE exactly matches the tactile MPC objective but does not necessarily represent actual distances between object positions and poses.The manually annotated centroid distance supplements MSE to capture positional accuracy.
B. Tactile Control Baseline
The baseline detects pressure centroids and moves in a straight line toward the target, using hand-engineered features and tuned step lengths. Deep tactile MPC consistently outperforms this baseline on the ball task and is more general across object mechanics.
- Tactile control baseline: The baseline detects object pressure centers, then commands straight-line movements toward the estimated target position with a tuned step length.Its features are manually engineered separately for the ball, joystick, and die tasks.
- Tactile control baseline: On both pressure-centroid distance and tactile-image MSE, deep tactile MPC consistently dominates the hand-designed baseline across 30 ball-task trajectories.Figure 7 reports threshold curves for both metrics, with the proposed method shown in orange and the baseline in blue.
- Tactile control baseline: The baseline is relatively strong for the ball because centroid location indicates the spherical ball’s position, but it frequently fails on joystick and die-rolling tasks.Its performance depends on object-specific geometry and mechanics.
- Tactile control baseline: Deep tactile MPC avoids manual tuning and object-specific mechanics by learning predictive dynamics from data, including how ball bearings move relative to sensor motion.The learned model captures basic physical properties without explicit hand-designed rules.
C. Manipulating a Ball, an Analog Stick, and a 20-sided Die
Deep tactile MPC enabled touch-only manipulation of a ball, analog stick, and die, outperforming a baseline on reported repositioning metrics. The method also achieved successful die-face changes despite complex contact dynamics.
- Touch-only control achieved all three manipulation tasks: ball repositioning, analog-stick control, and die rolling.The qualitative results cover manipulation using no sensory feedback beyond touch.
- For ball repositioning, the method consistently dominated the baseline across centroid-distance and image-MSE thresholds over 30 test trajectories.
- The method substantially outperformed the baseline on analog-stick repositioning under both reported metrics.Figure 8 evaluates centroid distance and tactile-image MSE over 15 trials.
- Deep tactile MPC achieved significantly lower median distance than the baseline for both ball rolling and analog-stick manipulation.The benchmark table also reports die-rolling success using the fraction of trials reaching the desired face.
- The authors conjecture that the larger die-rolling advantage reflects the baseline's difficulty handling slipping, sliding, and rolling across multiple surfaces.
VI. DISCUSSION AND FUTURE WORK
The paper presents high-bandwidth GelSight-based predictive control for tactile goal reaching and object repositioning. It reports successful touch-only manipulation, while identifying short horizons and single-finger hardware as boundaries on task complexity.
- The benchmark table covers ball rolling, analog-stick repositioning, and die rolling, using median L2 distance or desired-face success over 30 examples.
- A qualitative die example shows rolling from face 20 toward face 8 using real context frames and predicted tactile video frames.
- The method learns forward predictive models for high-bandwidth GelSight touch sensors to reposition objects and reach user-specified goals.
- A short-horizon controller is effective for simple servoing but limits tasks requiring multiple-object rearrangement or repeated complex finger gaits.
- With a single finger, the range of executable manipulation behaviors is limited to simple rearrangement.The paper proposes multi-finger tactile predictive models as a future direction for more complex in-hand manipulation.
VII. APPENDIX
The hardware setup mounts a custom GelSight sensor on a modified 3-axis CNC machine for controlled linear translation.
- A custom GelSight sensor is mounted on a modified 3-axis CNC machine that translates linearly along all three axes.
A. Autonomous Data Collection
Autonomous data collection uses resettable task environments, randomized three-axis actions, and recurrent tactile video prediction with real observations supplied as initial context frames.
- Task resets rely on environment dynamics, including a curved surface for the ball and springs for the analog stick.
- The experiments collect thousands of trajectories: 7400 for the ball, around 3000 for the analog stick, and 4500 for the die.
- Actions are parameterized as ± 6.0 mm changes along x, y, and z, collected at 1.5 Hz, while tactile MPC runs at about 1 Hz.Actions repeat for three time steps, with images recorded at every time step.
- The video-prediction model is a deep recurrent neural network that generates future images by transforming previous images.
- During the first three prediction steps, the recurrent model receives recent ground-truth observations called context frames.