Source-linked AI summary
Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, Sergey Levine
TL;DR
The paper asks how reinforcement learning can transfer skills between agents with substantially different morphologies and other domain structures. It learns invariant feature spaces from skills shared by both agents, then uses those spaces to guide transfer of new skills. In simulated manipulation, the method transfers skills across arms with different link counts and across torque-driven and tendon-driven actuation.
Problem
Skill transfer is difficult when agents differ in morphology, state spaces, action spaces, or dynamics, and direct state correspondences may not exist.
Method
The method learns mappings into an invariant feature space from shared proxy skills, then projects source executions into that space to shape target-agent exploration.
Results
The method transfers manipulation skills between simulated arms with different numbers of links and between torque-driven and tendon-driven arms.
Takeaways & Limitations
Shared skills can provide correspondences that let one agent transfer information about a new skill while ignoring aspects not shared across agents.
Takeaways & Limitations
The formulation assumes that source and target optimal-policy state distributions share structure after projection into a common feature space.
Abstract
from arXiv · showhide
People can learn a wide range of tasks from their own experience, but can also learn from observing other creatures. This can accelerate acquisition of new skills even when the observed agent differs substantially from the learning agent in terms of morphology. In this paper, we examine how reinforcement learning algorithms can transfer knowledge between morphologically different agents (e.g., different robots). We introduce a problem formulation where two agents are tasked with learning multiple skills by sharing information. Our method uses the skills that were learned by both agents to train invariant feature spaces that can then be used to transfer other skills from one agent to another. The process of learning these invariant feature spaces can be viewed as a kind of "analogy making", or implicit learning of partial correspondences between two distinct domains. We evaluate our transfer learning algorithm in two simulated robotic manipulation skills, and illustrate that we can transfer knowledge between simulated robotic arms with different numbers of links, as well as simulated arms with different actuation mechanisms, where one robot is torque-driven while the other is tendon-driven.
1 INTRODUCTION
The paper frames skill transfer between morphologically different agents as a reinforcement-learning problem and introduces invariant feature spaces learned from shared skills. It evaluates transfer across robot morphologies, state spaces, action spaces, and dynamics.
- Motivation: Observing another agent can accelerate skill learning even when the agents differ in morphology or execute the skill differently.The motivation draws on observations of human learning and differing physical capabilities.
- Problem: Existing state-space isomorphism methods often require domain knowledge, search over pairings, or simple low-dimensional environments.These approaches also assume that the two domains can be brought into correspondence.
- Method: The proposed method learns mappings from both agents’ states into an invariant feature space using skills previously acquired by both agents.A transferred skill is projected into the shared space, where the target tracks corresponding features through its own actions to obtain a shaped reward.
- Contributions: The work formulates multi-skill transfer, defines a common feature space, and learns mappings for transfer between agents with different morphologies.Deep neural networks represent potentially complex and nonlinear mappings.
- Evaluation: The evaluation transfers manipulation knowledge between simulated arms with different numbers of links and between torque-driven and tendon-driven arms.The experiments use two simulated robotic manipulation tasks.
2 RELATED WORK
Prior transfer-learning work includes shared state features, manually constructed mappings, manifold alignment, policy transfer, and learned feature spaces. The paper distinguishes its approach by learning shared representations for structurally different agents rather than directly transferring policies.
- Robotics and reinforcement learning: Earlier robotics methods transferred value functions or Q-functions through shared subsets, manually defined mappings, or manually constructed common feature spaces.These approaches generally rely on predefined correspondences or features.
- Robotics and reinforcement learning: Unsupervised manifold alignment assigns state pairings, but prior work assumes hand-designed features that provide distances between states.The cited work also targets transfer between robots with different configurations and action spaces.
- Machine learning: General transfer-learning research studies differing data distributions and multitask models, while multitask networks can leverage shared representations.These lines position shared representations within broader machine-learning transfer research.
- Deep learning: Policy-transfer methods reuse policy parameters across environments but have not addressed transfer between agents with structurally different state spaces.The cited examples use regularization or novel neural-network architectures.
- Distinction from prior work: This approach transfers information about shared skill structure through a common feature space while ignoring aspects that are not shared across robots.Direct policy transfer is typically impossible when morphological differences are substantial.
- Feature-space learning: Computer-vision research has learned feature spaces using pairwise similarity constraints and Siamese networks that bring paired images closer than unpaired images.These methods provide related precedents for learned representations and metric spaces.
3 PROBLEM FORMULATION AND ASSUMPTIONS
The paper formalizes transfer between distinct MDPs by learning an informative common feature space from shared proxy skills. The setup separates agent-specific from task-specific information and uses state correspondences to train mappings robustly.
- 3 PROBLEM FORMULATION AND ASSUMPTIONS: The source and target domains are MDPs with potentially different state spaces, action spaces, dynamics, and rewards, while their optimal-policy state distributions share projected structure.The assumption is illustrated with 3-link and 4-link arms performing the same end-effector task.
- 3.1 COMMON FEATURE SPACES: The common feature-space assumption requires mappings whose projected optimal-policy state distributions match across source and target domains.The mappings should preserve as much information as possible rather than collapse states into a constant.
- 3.2 LEARNING WITH MULTIPLE SKILLS: The agents are assumed to share previously learned skills that provide examples for constructing the common feature space.The paper uses shared prior skills instead of relying on direct exploration by both agents.
- 3.2 LEARNING WITH MULTIPLE SKILLS: Each domain’s state is divided into agent-specific and task-specific components to separate morphology-related information from task information.The proxy and test domains share agent-specific states and action spaces for each agent but differ in task-specific states and rewards.
- 3.2 LEARNING WITH MULTIPLE SKILLS: A proxy task learned by both agents supplies candidate correspondences, which can then support faster transfer of a skill known only by the source agent.In practice, multiple shared basic skills may serve as proxy tasks.
- 3.3 ESTIMATING CORRESPONDENCES FROM PROXY SKILL: The method learns a pairing of corresponding states from proxy-task executions and uses those pairs in a contrastive loss.The paper considers unsupervised alignment but focuses on simpler episodic-task approaches.
- 3.3.1 TIME-BASED ALIGNMENT: Time-based alignment pairs states visited at the same timestep when both agents perform an episodic task at roughly the same rate.This provides a simple approximate alignment for proxy-task trajectories.
- 3.3.2 ALTERNATING OPTIMIZATION USING DYNAMIC TIME WARPING: Alternating optimization improves robustness by repeatedly updating the common feature space and re-estimating correspondences with Dynamic Time Warping.It addresses sensitivity when the agents perform the task at different rates.
4 LEARNING COMMON FEATURE SPACES FOR SKILL TRANSFER
The method learns a shared embedding from a proxy skill using paired states, contrastive alignment, and reconstruction, then uses it to shape reinforcement learning for transferring a new skill. Because mappings need not be invertible and proxy pairings may be imperfect, target learning may still require adaptation and a decaying transfer weight.
- Learning and using the common feature space: The shared space is learned from a proxy task by aligning corresponding source and target states, then used to transfer a test skill.The target agent projects source executions into the shared space and receives guidance while learning its own policy.
- Learning the embedding functions: Contrastive alignment trains separate embedding functions to produce similar representations for paired states across domains.The functions are trained with a similarity objective over paired proxy states rather than requiring identical state representations.
- Learning the embedding functions: Decoder-based reconstruction prevents degenerate embeddings by encouraging the shared features to preserve domain information.Without reconstruction, both embeddings could collapse to a constant such as zero; decoders reconstruct each domain’s states from its embedding.
- Using the common embedding for knowledge transfer: The learned mappings establish an invariant space but need not be invertible, so the method matches trajectory distributions instead of directly mapping source states to target states.The target policy is learned with an additional reward term that encourages its embedded trajectories to follow the source trajectory distribution.
- Using the common embedding for knowledge transfer: Transfer reward shaping provides directed exploration guidance, especially in sparse-reward environments, while the target agent can learn morphological details from scratch.The transfer term is weighted by α relative to the task objective and supplies guidance through the source embedding.
- Using the common embedding for knowledge transfer: With imperfect state pairings, transfer reward can interfere once the target policy is already strong, motivating gradual reduction of α during learning.The authors use this schedule in the tendon-driven-arm experiment.
5 EXPERIMENTS
The experiments evaluate invariant-feature transfer across robots with different morphologies, actuation mechanisms, and image-based observations, emphasizing sparse-reward manipulation tasks. Across these settings, the method accelerates learning and outperforms or matches alternative transfer approaches.
- Transfer between robots with different numbers of links: The study evaluates transfer between 3-link and 4-link robots on sparse- or delayed-reward manipulation tasks, including button pressing.Proxy tasks include target reaching, block moving, and peg insertion.
- Transfer between robots with different numbers of links: The learned common feature space requires mapping joint angles and velocities to transferable end-effector information across robots with different link counts.The embedding functions are trained from successful proxy-task trajectories.
- Transfer between robots with different numbers of links: The proposed method reaches an almost perfect button-pressing success rate in 7 iterations, while CCA needs about 4 times more experience to reach 60% success.UMA and random projections perform much worse, and kernel CCA does not perform as well as the proposed method.
- Transfer between robots with different numbers of links: Direct state mapping provides limited transfer because morphologically different robots may lack complete state correspondence, whereas the common space retains transferable information.This comparison motivates learning partial correspondences rather than forcing full state-to-state alignment.
- Transfer between robots with different numbers of links: Learning embeddings from all three proxy tasks improves performance over embeddings learned from any single proxy task.The ablations use peg insertion, block pushing, or reaching individually, compared with their joint use.
- Transfer between torque-controlled and tendon-controlled manipulators: For torque-driven and tendon-driven 3-link arms, common-feature transfer enables substantially faster block-pulling learning than learning from scratch or using several alternative methods.The tendon-driven target fails from scratch, while CCA and direct mapping learn more slowly; quadratic-kernel CCA is competitive here but inconsistent across experiments.
- Transfer through image features: Raw-image transfer enables effective faster learning on the 4-link block-pushing task, whereas the no-transfer baseline fails because the reward is too sparse.Random projections and linear-kernel CCA show some transfer.
6 DISCUSSION AND FUTURE WORK
The method transfers skills between morphologically different agents by learning invariant feature spaces from shared skills. Experiments demonstrate transfer across robot morphologies and actuation mechanisms, while future work targets larger collections of behaviors and agents.
- Discussion: Shared skills let agents learn an invariant feature space that implicitly brings their domains into correspondence without assuming an explicit state-space isomorphism.A skill known to only one agent is mapped into this space so the other agent can transfer its shared structure.
- Discussion: The transferred shared structure can substantially accelerate the other agent’s learning of a skill known initially to only one agent.The approach uses a shared proxy task to learn the feature spaces and applies them to simulated manipulation skills.
- Discussion: Experiments transfer manipulation skills between simulated arms with different numbers of links and between torque-driven and tendon-driven arms.These results cover differences in morphology and actuation mechanism.
- Future work: Future work should handle many prior behaviors with varying degrees of similarity and decide which skills should be combined in a shared feature space.The paper contrasts sharing manipulation strategies across wheeled and legged robots with avoiding shared locomotion behaviors.
- Future work: A lifelong-learning extension could iteratively improve common feature spaces by transferring skills, incorporating them into the space, and transferring additional skills.Automatically selecting transfers to minimize total repertoire training time remains an open direction.
7 APPENDIX
The appendix uses trajectory-centric reinforcement learning for efficient robotic policy learning and permits other reinforcement learning methods or human demonstrations. Sparse target-domain rewards are used to assess the transfer reward’s detailed guidance.
- Appendix: The implementation uses time-varying linear-Gaussian policies trained with trajectory-centric reinforcement learning.The method is described as efficient and suited for robotic learning, despite producing simple policies.
- Appendix: The same reinforcement learning algorithm provides solutions in the source domain, although any suitable reinforcement learning method or human demonstrations could be used.This makes the transfer framework independent of the particular policy-learning algorithm used in the experiments.
- Appendix: Sparse reward functions in the target domain are used to evaluate whether the transfer reward provides detailed guidance.Original skills are generated in the source and proxy domains for training and transfer.