Hardware Conditioned Policies for Multi-Robot Transfer Learning

1 Introduction

• Millions of training examples may be needed even for simple actions, and hardware-specific policies can require separate training for each robot.The paper identifies both high sample complexity and poor cross-hardware generalization as obstacles to applying learned policies across robots.Source: §1 · p. 1
• Hardware differences in degrees of freedom, kinematic design, and dynamics prevent labs from pooling data effectively, while prior dynamic randomization does not address kinematics and degrees of freedom.This is especially relevant when robots are trained under torque control.Source: §1 · p. 1
• HCP conditions a universal policy on a hardware vector, using either explicit kinematic structure or a learned hardware embedding, and reports zero-shot transfer and sample-efficient fine-tuning.The introduction reports that learned embeddings can perform comparably to using full kinematics and dynamics information.Source: §1 · p. 2

Transfer in Robot Learning

• Transfer between different robots has received limited attention; Devin et al. transfer planar policies with separate robot- and task-specific modules, whereas this work conditions a universal policy on robot hardware.The paper says its hardware-conditioned approach is orthogonal to using a graph neural network as the policy class [23].Source: §2 · p. 2

Robust Control and Adaptive Control

• Dynamics randomization [4] can support generalization across dynamics, but the cited randomization work [4] uses position control, while this paper uses torque control and transfers between kinematically different agents.The paper also characterizes its method as model-free and not assuming linear system dynamics.Source: §2 · p. 2

System Identification

• System identification estimates unknown physical parameters or addresses model inconsistency; prior online identification [31] feeds predicted factors to the policy but required many samples in simple simulated domains.The paper contrasts this with hardware augmentation that it says is simple and independent of the policy class or DRL algorithm.Source: §2 · pp. 2–3

3 Preliminaries

• The problem is modeled as a fully observable continuous-space Markov decision process with states, actions, transitions, rewards, initial-state distribution, and discount factor.The objective is to find a policy maximizing expected return.Source: §3 · p. 3
• On-policy methods optimize the policy used during exploration, while off-policy methods can optimize using data collected by a different behavior policy.PPO is given as an on-policy example and DDPG as a model-free off-policy actor-critic algorithm for continuous actions.Source: §3 · p. 3
• Hindsight experience replay improves sparse-reward off-policy learning by treating an achieved goal as though it had been the requested goal.Repeated goal substitution is described as helping the agent learn to reach the specified goals.Source: §3 · p. 3

4 Hardware Conditioned Policies

• HCP represents robot hardware with a vector and conditions policy decisions on both the environment state and hardware characteristics.The considered properties include kinematics and dynamics; geometry and actuation design are left for future work, and dynamics may be unavailable or change over time.Source: §4 · p. 3

4.1 Explicit Encoding

• HCP-E explicitly encodes kinematic structure using the relative poses of consecutive joints, because end-effector information alone cannot capture how the kinematic chain affects behavior.Each relative pose combines a 3D displacement and a three-component Euler rotation vector; the resulting poses are concatenated into the hardware vector.Source: §4.1 · pp. 3–4

4.2 Implicit Encoding

• HCP-I learns a hardware embedding alongside the policy when dynamics matter and are difficult to measure directly.The embedding and policy parameters are updated through policy optimization, with gradients also updating the hardware vector.Source: §4.2 · p. 4

4.3 Algorithm

• HCP training samples a robot from a pool, augments states with its hardware vector, optimizes the policy, and also updates the vector when using implicit encoding.The paper describes the robot pool as containing varied kinematics and dynamics and notes PPO for dense-reward tasks and DDPG+HER for sparse-reward tasks.Source: §4.3 · pp. 4–5

5.1.1 Does HCP-E improve performance?

• Across multi-DOF reacher and peg-insertion experiments, HCP-E outperformed pooled-data DDPG+HER, which could not learn a common policy across robots without hardware information.The robot set varied joint damping, friction, armature, link mass, link length, kinematic structure, and degrees of freedom.Source: §5.1.1 · p. 6, Figure 3 · p. 6

5.1.2 Is HCP-E capable of zero-shot transfer to unseen robot kinematic structure?

• About 90% zero-shot success on robot type H and about 65% on 7-DOF type I with fixed-goal peg insertion show HCP-E transfers to unseen types where DDPG+HER barely succeeds.The experiments test on unseen robots with different link lengths and dynamics; the harder type H differs in joint configuration.Source: §5.1.2 · pp. 6–7, Table 1 · p. 7
• HCP-E achieved 75% success on the real Sawyer reacher task, while DDPG+HER could not move the arm close to the desired position.The evaluation averaged results over 20 random goal positions and included a policy trained using the Sawyer CAD model with randomized simulation dynamics.Source: §5.1.2 · p. 7, Figure 4 · p. 7
• On robot types H and I, fine-tuning pretrained HCP-E weights adapted quickly despite low zero-shot success; HCP-E also moved pegs closer to the hole than DDPG+HER.Figure 5 compares training from scratch with pretrained models for both robot types.Source: §5.1.2 · p. 8, Figure 5 · p. 8

5.2 Implicit Encoding

• For hopper locomotion, HCP-I performed comparably to HCP-E+Dyn and better than PPO; unlike explicit encoding, it can learn kinematics and dynamics information.HCP-E conditions only on kinematics, while HCP-E+Dyn uses ground-truth kinematics and dynamics.Source: §5.2 · p. 8, Figure 6 · p. 9
• Pretrained HCP-I hidden layers learned faster on 100 new hoppers than training from scratch, and its embedding varied smoothly with torso mass.The new-robot transfer fine-tunes policy parameters and estimates the unknown hardware representation.Source: §5.2 · pp. 8–9

6 Conclusion

• HCP-E supports zero-shot transfer when task policies do not heavily depend on dynamics, and fine-tuning can quickly adapt policies with low zero-shot success.Its explicit encoding represents kinematics, while HCP-I learns a hardware embedding for tasks where dynamics information improves learning.Source: §6 · p. 9
• HCP-I can perform on par with models using ground-truth kinematics and dynamics, without using either information source.The learned embedding is trained automatically through back-propagation.Source: §6 · p. 9

A.1 Algorithms

• The appendix gives on-policy PPO and off-policy DDPG+HER HCP algorithms, with robot hardware representations added to states and learned representations updated by gradient descent.The algorithms use PPO for dense-reward tasks and DDPG+HER for sparse-reward multi-goal tasks.Source: §A.1 · p. 13

B.1 Reacher and Peg Insertion

• Experiments cover reacher, peg insertion, and hopper; reacher represents a building block for manipulator tasks, while peg insertion tests control accuracy and robustness in torque-control transfer.The figure identifies the reacher start and target regions and marks joints in the reacher and peg-insertion robots.Source: §B.1 · p. 14, Appendix B · p. 14, Figure 7 · p. 14

B.1.1 Robot Variants

• Training uses nine robot types with different degrees of freedom and joint placements, with 5- and 6-DOF variants made by removing joints from a 7-DOF robot.Link lengths and dynamics parameters are varied, and damping samples are split evenly below 1 and at or above 1.Source: §B.1.1 · p. 14, Table 2 · p. 14, Figure 8 · p. 14
• Although training uses only robot types listed in Figure 2, the policy transfers directly to other new robots, as illustrated by the Fetch robot in Figure 9.Source: §B.1.1 · p. 14

B.1.2 Hyperparameters

• 86.30 ± 4.41% success on unseen Type J robots and 100% on Fetch are reported for the policy trained in Exp. V.The Type J result averages 10 trials of 100 robots each; the Fetch result comes from 10 tests.Source: Figure 9 · p. 15
• Reacher starts and targets are sampled from separate boxes, while peg-insertion experiments use a fully expanded starting pose and fixed or randomly moved hole positions.The reacher start box is 0.3m × 0.4m × 0.2m; its target box is 0.3m × 0.6m × 0.4m, located 0.2m below the initial region.Source: §B.1.2 · p. 15
• Reacher and peg insertion use an action-penalized reward, and success requires the point of interest to be within 0.02m of its target.State noise is uniformly sampled from [−0.02, 0.02] for joint angles and velocities.Source: §B.1.2 · p. 15

B.2 Hopper

• Hopper uses the OpenAI Gym reward design and PPO for its dense-reward setting, with SELU activations and the Adam optimizer.Its hyperparameters and parameter sampling ranges are summarized in Tables 4 and 5.Source: Table 4 · p. 15, §B.2 · p. 16

Appendix C Supplementary Experiments

• Table 5 is identified as a summary of hopper parameters.The supplied section roadmap does not specify the table's parameter values.Source: Table 5 · p. 16
• The supplementary experiments examine dynamics effects in manipulators and learning curves for 7-DOF robots with varied link lengths and dynamics.They also report HCP-E training details and performance on robots from training types with different link lengths and dynamics.Source: Appendix C · p. 16

• The comparison tested DDPG+HER with state-only, dynamics-augmented, and random-vector-augmented inputs on 7-DOF Type I robots with varied dynamics.The dynamics vector was scaled to [0, 1), and the random vector used the same size and range.Source: §C.1 · p. 16
• About 100% success was achieved on reacher and fixed-hole peg insertion using only joint angles and velocities as state.Figure 10 reports this result for DDPG+HER.Source: §C.1 · p. 16, Figure 10 · p. 16
• Adding ground-truth dynamics or a same-sized random vector to the inputs slowed learning compared with using only joint angles and velocities.The experiments used 100 training robots and compared the three input variants across varied dynamics.Source: §C.1 · p. 16
• A random input vector did not prevent generalization to new test robots, indicating that the policy learned to ignore the augmented input.Source: §C.1 · p. 16

C.2 How robust is the policy network to changes in dynamics?

• The peg-insertion stress test varied damping beyond the training range while sampling other dynamics parameters as in Table 2.Training damping was sampled from [0.01, 2), and the evaluation used 100 testing robots.Source: §C.2 · p. 17, Figure 11 · p. 17
• 100% success held for damping ranges [2, 10), [10, 20), and [20, 30); it fell to 85% for [30, 40) and 47% for [40, 50).The model was trained with damping sampled from [0.01, 2); Figure 11 and Table 6 report the test results.Source: §C.2 · p. 17, Figure 11 · p. 17, Table 6 · p. 17
• As damping increases, peg insertion takes more time steps, and high damping can prevent movement when joint torque limits bind.Source: §C.2 · p. 17, Figure 11 · p. 17

C.3 Learning curves for 7-DOF robots with different link length and dynamics

• For 7-DOF robots with different link lengths and dynamics, HCP-I and HCP-E achieved much higher success rates on both tasks than vanilla DDPG+HER.The passage notes that dynamics parameters are usually difficult to access on real robots, making their use impractical in robotics applications.Source: Figure 12 · p. 18

• The experiments used leave-one-out training on eight robot types and evaluated 100 test robots of those types with different link lengths and dynamics.The figure also covers reacher and peg insertion, including random hole positions within a 0.2 m box.Source: Figure 13 · p. 18
• Tables 7–14 report universal-policy behavior on trained robot types with different link lengths and dynamics, while Table 1 is cited for HCP-E testing on a novel robot type.Source: §C.4 · p. 18
• The trained HCP-E model achieved a 6.70 success rate on robot type D in peg insertion with random hole positions.The authors attribute this result to limited joint displacements and degrees of freedom; type D lacks J4 and J5, which they describe as crucial for peg insertion.Source: §C.4 · p. 19, Table 12 · p. 20