Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry

Siyuan Dong; Devesh K. Jha; Diego Romeres; Sangwoon Kim; Daniel Nikovski; Alberto Rodriguez

Source-linked AI summary

Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry

Siyuan Dong, Devesh K. Jha, Diego Romeres, Sangwoon Kim, Daniel Nikovski, Alberto Rodriguez

arXiv:2104.01167v1Apr 2, 2021cs.RO

TL;DR

The paper addresses insertion of general objects with unknown geometry, where contact configurations are difficult to observe and align. It uses an episodic reinforcement-learning policy with tactile feedback, curriculum training, and tactile-flow representation, achieving strong generalization to novel object geometries while comparing sensing and learning choices.

Problem
Insertion with unknown object geometry is challenging because contact configurations and full pose errors are not directly observable from a single force or tactile sensing instant.
Method
The paper learns an episodic reinforcement-learning policy that alternates insertion attempts and alignment corrections using tactile image sequences, curriculum learning, and tactile-flow representations.
Results
The RL policy with curriculum training and tactile flow generalizes to novel object geometries, while tactile sensing generalizes better than force/torque sensing despite the latter learning more efficiently.
Takeaways & Limitations
Tactile-flow reinforcement learning provides a general insertion policy across object geometries without prior geometric knowledge, outperforming the evaluated supervised and force/torque alternatives in generalization.

Abstract

from arXiv · show

Object insertion is a classic contact-rich manipulation task. The task remains challenging, especially when considering general objects of unknown geometry, which significantly limits the ability to understand the contact configuration between the object and the environment. We study the problem of aligning the object and environment with a tactile-based feedback insertion policy. The insertion process is modeled as an episodic policy that iterates between insertion attempts followed by pose corrections. We explore different mechanisms to learn such a policy based on Reinforcement Learning. The key contribution of this paper is to demonstrate that it is possible to learn a tactile insertion policy that generalizes across different object geometries, and an ablation study of the key design choices for the learning agent: 1) the type of learning scheme: supervised vs. reinforcement learning; 2) the type of learning schedule: unguided vs. curriculum learning; 3) the type of sensing modality: force/torque (F/T) vs. tactile; and 4) the type of tactile representation: tactile RGB vs. tactile flow. We show that the optimal configuration of the learning agent (RL + curriculum + tactile flow) exposed to 4 training objects yields an insertion policy that inserts 4 novel objects with over 85.0% success rate and within 3~4 attempts. Comparisons between F/T and tactile sensing, shows that while an F/T-based policy learns more efficiently, a tactile-based policy provides better generalization.

I. INTRODUCTION

The paper addresses insertion under unknown geometry by using tactile feedback to correct misalignment without explicitly recovering contact locations. It studies an episodic RL policy and ablates learning scheme, curriculum, sensing modality, and tactile representation.

Unknown geometric mismatch can create unexpected contacts and complex interlocking configurations during insertion.
Existing contact-localization techniques often require object geometry or support only limited contact configurations.
The proposed feedback mechanism corrects object-hole misalignment without geometry models or explicit contact-location inference.
The key challenge is partial observability: a single force or tactile observation does not fully reveal the contact state.
The study evaluates one policy across multiple objects and environments while comparing supervised learning, RL, curriculum training, F/T sensing, and tactile representations.
Tactile flow is reported to generalize better than tactile RGB across object and grasp geometry, while tactile-RL generalizes better than F/T-based policies.

II. RELATED WORK

Prior insertion methods include passive compliance, model-based force feedback, and RL, but many depend on geometry, restricted object types, or positional and force-torque state. This paper instead trains an RL policy directly from vision-based tactile feedback in a more varied insertion setting.

Classical passive-compliance methods succeed for chamfered holes when initial contact begins on the chamfer.
Model-based force-feedback methods estimate peg pose but require object models or remain constrained to a single object type.
Alignment errors can map ambiguously to sensed forces, making direct inversion impossible in many contact situations.
Earlier RL approaches primarily use robot positional state and force-torque measurements, while this work uses tactile feedback for insertion guidance.
The paper extends supervised tactile-error estimation to a more challenging scenario involving different objects and environments.

III. METHODOLOGY

The methodology uses high-resolution tactile sensing to capture contact-induced object motion and represents that information through tactile flow. Distinct flow patterns provide cues about external contact configurations during insertion.

The method uses two high-resolution tactile sensors to capture subtle object rotation signals during contact.
Figure 2 visualizes tactile-image sequences and marker-field displacement during contact between a cylindrical object and an environmental edge.
Different contact locations generate different tactile-flow patterns during insertion.
When the object contacts an edge perpendicular to the sensor surface, in-plane object rotation appears in the captured tactile pattern.

B. Deep RL Controller

The paper formulates insertion as a sequential decision problem because tactile observations may not reveal the full pose error at every step. RL can therefore optimize insertion outcomes over sequences of corrective actions rather than greedily estimating pose error.

Supervised tactile pose-error estimation can fail in complex environments such as rectangular holes because partial observability can make the controller diverge.
The RL controller takes tactile image sequences as input, outputs operational-space robot motions, and receives rewards tied to pose correction and insertion success.
RL can seek delayed-reward, non-greedy action sequences that insert the object when full pose error is not observable from tactile signals.

C. Curriculum Learning

The policy is trained on real-robot insertion with curriculum learning that progressively increases environment and object complexity. This staged design improves data efficiency while addressing increasingly constrained contact conditions.

Curriculum Learning: Real-robot training avoids inaccurate tactile-contact simulation, while curriculum learning makes reinforcement-learning data collection more efficient.The policy is trained on a physical robot because tactile signals are difficult to simulate accurately.
Curriculum Learning: The curriculum progresses from line wall to corner wall, U wall, and hole environments, adding one constraint at each stage.The sequence incrementally introduces contact-direction reasoning, overshoot avoidance, and multi-axis step-size control.
Curriculum Learning: Training uses four object shapes—cylinder, hexagonal cylinder, elliptical cylinder, and cuboid—to promote generalization across geometries.The cuboid is hardest because small yaw misalignment can block insertion.
Curriculum Learning: Tactile-signal representation matters because high-dimensional tactile images and few training objects can cause overfitting.Surface texture may be irrelevant to insertion, whereas contact-related information is more useful for generalization.

IV. EXPERIMENTAL DETAILS

Experiments use a robot, gripper, GelSlim tactile sensors, multiple insertion environments, and controlled pose perturbations. Each episode tests vertical insertion under randomized object and alignment conditions.

Experimental Setup: The setup includes a 6-DoF robot arm, parallel-jaw gripper, two GelSlim sensors, an F/T sensor, four objects, and four insertion environments.The environments are line walls, corner walls, U walls, and holes.
Experimental Protocol: Pose perturbations range from -6 to 6 mm in x and y and from -10° to 10° in yaw before insertion.These perturbations emulate residual error after vision-based hole localization.
Experimental Protocol: Four tactile contact configurations distinguish in-plane versus out-of-plane rotation and differing relative vertical sensor displacements.The first two cases involve perpendicular edges; the last two involve edges parallel to the sensor plane.
Experimental Protocol: Resetting varies grasp force and object height to reduce overfitting to particular heights or local tactile-image features.The mechanism also supports controlled object resetting under noisy initial conditions.
Experimental Protocol: Each episode grasps a randomly selected object, attempts vertical insertion, and treats an untriggered slip detector as successful insertion.A triggered slip detector indicates that the insertion is blocked by the environment.

C. Deep RL Policy

The deep-RL policy uses TD3 with tactile-image sequences as state and continuous pose corrections as actions. Training combines reward shaping, curriculum learning, and supervised bootstrapping to improve sample efficiency.

Deep RL Policy: The policy uses 24 tactile images from two sensors as the state and outputs continuous gripper-frame displacements in x, y, and θ.The state contains 12 images per sensor captured during contact, while actions specify Δx, Δy, and Δθ.
Deep RL Policy: TD3 is selected as an off-policy algorithm to improve sample efficiency in tactile-policy training.The actor observes tactile sequences, while the critic uses current contact error and the actor action.
Deep RL Policy: The reward combines contact-error reduction, a penalty for each insertion attempt, and an additional reward for successful insertion.The success term is activated only when insertion succeeds.
Deep RL Policy: Training uses curriculum learning and stops when recent rewards have sufficiently high mean and low standard deviation.The stopping criterion is evaluated over the last 30 episodes.
Deep RL Policy: The actor is initialized from 300 supervised examples, then frozen for the first 50 reinforcement-learning episodes.This bootstrapping helps populate the replay buffer with positive examples before critic-driven updates.

D. Supervised Learning Policy

The supervised baseline predicts object-hole misalignment from tactile image sequences and iteratively applies the opposite correction. It uses the same actor architecture as the reinforcement-learning policy for comparison.

Supervised Learning Policy: The supervised policy maps tactile image sequences to estimated contact error in x, y, and θ.It is trained with mean-squared error using data and misalignment labels collected during reinforcement-learning training.
Supervised Learning Policy: During execution, the baseline applies the opposite of its predicted error to minimize misalignment iteratively.Its maximum number of insertion attempts is 15.
Supervised Learning Policy: The supervised baseline uses the same architecture as the reinforcement-learning actor for a fair comparison.The actor model combines CNN, RNN, and MLP components, with marker flow used only for visualization in the cited figure.

E. RL policy with F/T sensor

The F/T-sensing policy is evaluated across training and novel objects, with performance varying by object geometry and initial pose error. Its main failure mode is difficulty distinguishing rotational misalignment, especially when translation and rotation errors are both large.

Evaluation: The evaluation uses 250 insertion experiments per object under uniformly sampled initial pose errors, measuring success rate and attempts over successful cases.The x and y errors range from -5mm to 5mm, while θ ranges from −10° to 10°.
Training objects: The RL* policy achieves 90% success on the cuboid and over 97% on the other three training objects.The cuboid requires around 5–6 attempts, while the other training objects require 2–3 attempts.
Novel objects: On novel objects, success exceeds 96% for the small bottle and phone charger but drops to 80% for the big bottle.The big bottle’s chamfered bottom edge changes how the object rotates during insertion.
Failure cases: The policy’s failures most often occur when both translational and rotational components of the initial pose error are large.In these cases, the tactile sensors may capture too little rotation, causing the policy to choose an opposing correction and diverge.

B. Baseline Comparison

The baseline comparisons show that curriculum-trained RL with tactile flow outperforms supervised learning and unguided RL on generalization, while tactile flow avoids the overfitting seen with RGB input. F/T sensing performs well on round objects but is less reliable for rotation-sensitive geometries.

Supervised vs. reinforcement learning: The supervised policy performs worse than RL* on training objects and generalizes poorly, barely inserting the cuboid during testing.Only the small bottle exceeds 70% success among novel objects, while ambiguous tactile signals can make the supervised controller output zero and get stuck.
Curriculum training: Unguided RL achieves only 30% average success on new objects when trained directly in the hole environment.This comparison evaluates curriculum training using the same bootstrap and tactile-flow input as RL*.
Tactile representation: RGB-input RL performs as well as RL* on training objects but drops dramatically on novel objects, consistent with overfitting to object-specific features.The effect is especially pronounced for the big bottle and paper box, where the policy repeatedly outputs the same action.
F/T versus tactile sensing: F/T sensing achieves almost 100% success on cylindrical objects but falls to around 55% on three cuboid-shaped objects.The F/T policy localizes x and y contact errors but has trouble distinguishing rotational error, whereas tactile sensing also observes object motion during contact.

VI. CONCLUSION

The paper proposes geometry-free RL insertion using tactile feedback, curriculum training, and tactile flow, and reports advantages over several tactile baselines and F/T sensing for generalization.

Conclusion: The proposed policy uses reinforcement learning, curriculum training, and tactile flow from two GelSlim sensors without prior object-geometry knowledge.The paper compares this configuration with supervised learning, RL without curriculum, RGB-input RL, and F/T sensing.