Source-linked AI summary
TaRL: Learning General and Physical Rewards from Tactile Demonstrations
Po-Yi Wu, Dao-Jan Chang, Shang-Ya Hsiao, Hong-Ming Chen, Yu-Cheng Su, Tsung-Wei Ke
TL;DR
Contact-rich manipulation needs rewards for physical interactions that visual observations can miss. TaRL regresses task progress from tactile deformation maps and uses the learned reward to shape downstream reinforcement learning. It improves learning and success across simulated and real-world tasks, with gains on nut threading and cube pickup.
Problem
Visual reward learning conditions on visual observations and cannot capture contact-level goals such as firm grasps and directed forces when they are occluded.
Source: §I · p. 1Method
TaRL regresses task-completion progress from tactile deformation-map sequences in successful and failed demonstrations for use as a downstream RL shaping reward.
Source: §I · p. 2, B. Tactile Reward Learning · p. 3Results
TaRL improves downstream RL sample efficiency and final success across four simulated tasks; nut-threading success rises from 34% to 56%, and cube pickup from 37% to 97%.
Source: A. Does TaRL improve downstream RL? · p. 5, E. Real-World Experiments · p. 7, §V · p. 8Takeaways & Limitations
Tactile rewards capture contact-level information that visual rewards miss, and combining tactile and visual rewards yields the best policies.
Source: §V · p. 8TaRL is trained one task at a time, so each new task requires its own tactile demonstrations and reward model.
Source: §V · p. 8
Abstract
from arXiv · showhide
Contact-rich manipulation requires robots to sequence precise contacts, maintain stable grasps, and apply directed forces. Reinforcement learning (RL) can acquire such behaviors automatically, but its performance hinges on reward design: sparse rewards reduce the learning efficiency, while dense rewards are hard to specify. Visual reward learning addresses this by inferring rewards from action-free demonstrations. Because it conditions only on visual observations, it fails to capture rewards beyond visual goals. We propose Tactile Reward Learning (TaRL), a framework that learns rewards from tactile demonstrations. TaRL takes a sequence of tactile deformation maps as input, and regresses task-completion progress from both successful and failed demonstrations. Because TaRL captures local robot-object interaction, it provides informative feedback to learn firm grasps and correctly directed forces; meanwhile, it is robust to changes in scene layout such as object position. We evaluate TaRL on four manipulation tasks in simulation and two in the real world. Used as a shaping reward, it substantially improves both sample efficiency and final success rate, raising success on Nut threading from 34% to 56% in simulation and on cube pickup from 37% to 97% in the real world. Combining tactile with visual rewards improves performance further. TaRL also generalizes across object instances: trained on box placement and directly deployed to can placement, it significantly improves policy learning on the new task. Project page is available at https://embodiedai-ntu.github.io/tarl.
I. INTRODUCTION
Contact-rich manipulation needs precise contacts, stable grasps, and directed forces, but reward design makes reinforcement learning difficult. TaRL learns task-progress rewards from tactile demonstrations to address physical goals visual rewards miss, and improves downstream learning across simulation and real-world tasks.
I. INTRODUCTION
- Reward design bottlenecks reinforcement learning because sparse rewards make learning inefficient, while dense rewards are difficult to specify and can induce unintended behavior.Contact-rich tasks also require orchestrating precise contacts, stable grasps, and directed forces.Source: §I · p. 1
- Visual rewards miss contact-level goals such as grasp firmness and force direction because these interactions can be occluded from view.Tactile sensing directly encodes this physical information.Source: §I · pp. 1–2
- TaRL learns task-progress rewards from tactile deformation-map sequences in successful and failed demonstrations, without ground-truth states, visual observations, or action annotations.The pre-trained reward model supplies dense signals to downstream RL without online fine-tuning.Source: §I · p. 2
- TaRL is evaluated on four simulated tasks and two real-world tasks, and improves training efficiency while generalizing to unseen object positions and instances.Across experiments, tactile demonstrations are collected at one object position while policies are evaluated at disjoint positions.Source: §I · p. 2
- Combining TaRL with visual reward learning produces more complete reward signals and significantly enhances RL training efficiency.The paper describes tactile reward learning as orthogonal to existing visual reward learning.Source: §I · p. 2
II. RELATED WORK
Prior reward-learning approaches ground rewards in scene appearance, leaving contact-centric variables largely unobserved; TaRL instead learns rewards from tactile demonstrations. It uses task-progress regression and applies predicted tactile rewards in downstream reinforcement learning.
II. RELATED WORK
- Existing visual reward methods ground rewards in scene appearance, leaving contact-centric variables such as grasp stability and applied force largely unobserved.Prior tactile work has mainly used touch as policy input or for prediction and planning; See-to-Touch derives reward visually.Source: §II · p. 2
III. METHOD
- TaRL models reward as task-completion progress from tactile signals that evolve from initial contact through stable grasp to directed force application.Its tactile rewards do not depend on scene appearance, supporting robustness to layout changes and transfer across tasks.Source: §III · p. 2
A. Problem Formulation
- Sparse rewards indicate success or failure for the whole task or a manipulation stage, while the ground-truth dense reward is unknown and auxiliary reward engineering is laborious.This leaves sparse supervision insufficient for the task.Source: A. Problem Formulation · p. 2
III. METHOD
- The TaRL pipeline trains on successful, rewound, and failed demonstrations, maps tactile deformation sequences to task-progress rewards, and supplies dense rewards to downstream RL.The framework builds on a common reward-learning formulation and leaves more advanced designs for future work.Source: §III · p. 2, Figure 2 · p. 3
A. Problem Formulation
- TaRL focuses on progress regression, where a reward model predicts task progress from the trajectory observed so far rather than conditioning on a goal image.The cited formulation predicts progress as t/T over time horizon T.Source: A. Problem Formulation · p. 3
B. Tactile Reward Learning
TaRL adapts a trajectory-conditioned reward-learning framework to tactile videos, using demonstrations and causal progress prediction to provide dense rewards. Its preprocessing and training examples are designed to learn progress from relative tactile patterns and both successful and failed demonstrations.
B. Tactile Reward Learning
- TaRL replaces ReWiND’s RGB-video conditioning with tactile videos while retaining its trajectory-conditioned formulation and model architecture.The stated aim is to study tactile feedback as an alternative reward signal rather than introduce a new model design.Source: B. Tactile Reward Learning · p. 3
- The demonstrations pair full tactile observation sequences with binary task outcomes, collected from simulated RL rollouts or real-world teleoperation.Simulation rollouts span failures, near-misses, and successes; real-world demonstrations include successful and failed attempts.Source: B. Tactile Reward Learning · p. 3
- TaRL uses paired 3D deformation maps, which capture contact-surface pressure and shear forces, as its tactile observations.Each map represents one gripper finger’s deformation.Source: B. Tactile Reward Learning · p. 3
- A causal transformer processes encoded tactile observations and predicts task progress at each timestep, with the latest sampled token providing the tactile reward.The model uniformly subsamples 16 timesteps and pads shorter histories with the most recent frame.Source: B. Tactile Reward Learning · p. 3
- Per-sequence normalization scales deformation maps by their largest absolute entry, encouraging reward prediction from relative interaction patterns rather than absolute deformation values.The scaling factor has a minimum of 10^-6; the passage links this preprocessing to generalization across episodes, object instances, and tasks.Source: B. Tactile Reward Learning · p. 4
- Training regresses progress targets from successful demonstrations, rewound successful videos, and failed demonstrations, respectively targeting t/T, (i −t)/T, and 0.Rewound examples concatenate a forward prefix and reversed suffix at a random split point.Source: B. Tactile Reward Learning · p. 4
C. Downstream RL with TaRL
After offline training, TaRL supplies dense tactile shaping rewards during downstream RL without further fine-tuning. These rewards can be combined with environment and visual rewards, with simulation and real-world experiments using different RL algorithms.
C. Downstream RL with TaRL
- After offline training, TaRL is used directly during downstream rollout collection without further fine-tuning.The model subsamples tactile maps from the current episode’s observed history, padding with the latest frame when needed.Source: C. Downstream RL with TaRL · p. 4
- The total reward combines the environment task reward with weighted tactile and visual shaping rewards.The task reward omits terms encouraging stable grasps or precise contact; α and β weight tactile and visual rewards.Source: C. Downstream RL with TaRL · p. 4
- Simulation policies use PPO, while real-world policies use offline IQL.Source: C. Downstream RL with TaRL · p. 4
IV. EXPERIMENTS
The experiments evaluate whether TaRL improves downstream RL, compares with visual rewards, generalizes to new object instances, and supports real-world RL. Policies are assessed across contact-rich tasks and at object positions disjoint from those used for demonstrations.
IV. EXPERIMENTS
- The experiments also examine comparison with visual rewards, transfer to novel object instances, whether learned tactile rewards are necessary, and real-world RL.Source: §IV · p. 4
- Demonstrations are collected at one object position, while downstream policies are evaluated at disjoint positions to test generalization to out-of-distribution configurations.Reward quality is measured against ground-truth progress on held-out trajectories; downstream success is averaged over three seeds within a fixed interaction budget.Source: §IV · p. 4
- The evaluation covers four simulation tasks, including box placement and contact-intensive peg insertion, gear assembly, and nut threading.The setup adds reach and grasp before insertion, threading, or placement, rather than initializing objects pre-grasped.Source: §IV · pp. 4–5
- TaRL substantially enhances downstream RL training efficiency and task success in simulation.Fig. 4 compares policies trained with and without TaRL.Source: Figure 4 · p. 5
A. Does TaRL improve downstream RL?
TaRL as a shaping reward improves downstream RL and distinguishes successful from failed trajectories, including at unseen object positions.
A. Does TaRL improve downstream RL?
- The simulation evaluation trains TaRL on 200 successful and 800 failed demonstrations per task.Source: A. Does TaRL improve downstream RL? · p. 5
- On nut threading, TaRL raises task success from 34% to 56% while substantially improving RL learning efficiency and final success on most tasks.The environments omit rewards for stable grasp and precise contact; visual rewards are disabled in this comparison.Source: A. Does TaRL improve downstream RL? · p. 5
- TaRL predicts higher rewards for successful than failed trajectories and generalizes to novel object configurations.The Fig. 5 comparison averages predicted reward at each time step for seen and unseen object positions.Source: A. Does TaRL improve downstream RL? · p. 5, Figure 5 · p. 5
B. How does TaRL compare with visual reward model?
TaRL and visual reward learning provide complementary signals: visual rewards capture spatial relationships, while tactile rewards remain informative across contact phases and object positions.
a) OOD generalization
- With multiple-position training, ReWiND’s successful-versus-failed reward gap falls from 0.5 to 0.1 on unseen object configurations.The single-position visual model handles seen configurations but fails to generalize to unseen positions.Source: B. How does TaRL compare with visual… · p. 5, Figure 7 · p. 5
- Across peg insertion, gear assembly, and nut threading, tactile feature distributions remain similar between ID and OOD positions, unlike visual feature distributions.The analysis visualizes features as 2D t-SNE vectors; the authors relate tactile features to local robot-object interaction and robustness to scene-layout changes.Source: B. How does TaRL compare with visual… · p. 6, Figure 6 · p. 6
b) Rewards contributed in individual contact phases
- In gear assembly, visual reward plateaus when the end-effector reaches the target, whereas tactile reward grows through distinct contact phases until success.Visual reward emphasizes robot-scene spatial relationships; tactile reward provides feedback during local interactions.Source: B. How does TaRL compare with visual… · p. 6, Figure 8 · p. 6
c) Complementary reward shaping
- Combining tactile and visual rewards accelerates downstream RL training on peg insertion, gear assembly, and nut threading.The authors report substantially increased task success for peg insertion.Source: B. How does TaRL compare with visual… · p. 6
Zero-shot generalization to unseen object instances
- TaRL trained on box placement transfers to can placement without extra training and improves downstream RL efficiency on that task.The predicted rewards distinguish successful from failed trajectories on the unseen task.Source: Figure 9 · p. 6
C. Does TaRL generalize to novel object instances?
TaRL trained on box placement transfers to can placement without extra training, distinguishing successful from failed trajectories and improving downstream RL efficiency.
C. Does TaRL generalize to novel object instances?
- Without extra training, TaRL trained on box placement distinguishes successful and failed can-placement trajectories and significantly improves downstream RL efficiency.Source: C. Does TaRL generalize to novel… · p. 6
D. Is a learned tactile reward necessary?
Experiments find that learned tactile rewards still improve RL when the actor and critic receive tactile observations, and outperform handcrafted tactile rewards in nut threading.
D. Is a learned tactile reward necessary?
- TaRL still substantially improves RL learning efficiency and task success when the actor and critic take tactile observations as input.The tested setup encodes tactile observations into a latent vector and concatenates it with environment-state vectors.Source: D. Is a learned tactile reward… · pp. 6–7
- On nut threading, RL using TaRL learns more efficiently than RL using a handcrafted tactile grasp reward.The handcrafted reward signals when the target object is grasped.Source: D. Is a learned tactile reward… · p. 7, Figure 11 · p. 7
- The handcrafted reward cannot distinguish good from bad behavior after grasping, limiting feedback across contact phases and leading to suboptimal RL performance.Source: D. Is a learned tactile reward… · p. 7
E. Real-World Experiments
In real-world cube pickup and peg insertion, policies trained with TaRL rewards outperform those trained with sparse rewards alone. The experiments use teleoperated demonstrations and offline RL across positions distinct from TaRL’s training position.
E. Real-World Experiments
- TaRL uses 40 successful and 40 failed teleoperated trajectories from one object position; offline policy data uses 90 successful and 90 failed trajectories at different positions.The fixed policy dataset receives human-labeled sparse task-completion rewards and dense task-progress rewards from TaRL.Source: E. Real-World Experiments · p. 7
- Policies are trained with IQL across three random seeds and evaluated by final task success, with peg insertion reported separately for pickup and insertion phases.Source: E. Real-World Experiments · p. 7
- Cube pickup success rises from 37% to 97% with TaRL; peg pickup and insertion gain 45% and 10% absolute performance, respectively.Policies with tactile rewards outperform sparse-reward policies consistently across all evaluated tasks.Source: E. Real-World Experiments · p. 7, Figure 13 · p. 7
V. CONCLUSION
TaRL learns task progress from tactile deformation-map sequences, capturing contact structure that visual rewards miss. It improves downstream RL across simulated tasks and in real-world cube pickup, while cross-task reward-model transfer without retraining remains unresolved.
V. CONCLUSION
- TaRL regresses task progress from tactile deformation-map sequences, capturing contact-rich physical structure missed by visual rewards.The conclusion argues that combining visual and tactile rewards yields the best policies because the modalities inform different task phases.Source: §V · pp. 7–8
- Across four simulated tasks, TaRL improves downstream RL sample efficiency and final success, generalizes to unseen positions and object instances, and raises real-world cube pickup success from 37% to 97%.It also outperforms tactile policy inputs and handcrafted tactile rewards.Source: §V · p. 8
- TaRL is trained one task at a time, requiring new tactile demonstrations and a reward model for each task; transfer without retraining remains open.Source: §V · p. 8