Source-linked AI summary
panda-gym: Open-source goal-conditioned environments for robotic learning
Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, Liming Chen
TL;DR
Sparse-reward robotic learning remains difficult and interaction-intensive, motivating simulated environments for complex tasks. panda-gym provides five goal-conditioned Panda tasks in an extensible PyBullet package, with baselines showing that performance varies by task: DDPG solves reach and push but not stacking within the reported budget.
Problem
Current reinforcement-learning algorithms struggle with sparse-reward tasks and may require many interactions, while complex tasks require simulation for learning.
Source: §1 · p. 1Method
The paper presents five Multi-Goal RL environments for the Panda in PyBullet, with modular robot-task design and sparse rewards by default.
Source: §2 · p. 1, §2.3 · p. 3, §3 · p. 3Results
DDPG reaches 100% success on reach after 10^4 and push after 3 × 10^4 timesteps, but stacking remains near zero after 1.6 × 10^6; clipped double-Q ablation improves TD3 and SAC results in multiple environments.
Source: §4 · p. 4Takeaways & Limitations
The package provides an open-source way to evaluate reinforcement-learning algorithms on complex robotic tasks, with some tasks solved and others remaining unsolved.
Source: §5 · p. 4The presented algorithms do not solve stacking within 1.6 × 10^6 timesteps.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper presents panda-gym, a set of Reinforcement Learning (RL) environments for the Franka Emika Panda robot integrated with OpenAI Gym. Five tasks are included: reach, push, slide, pick & place and stack. They all follow a Multi-Goal RL framework, allowing to use goal-oriented RL algorithms. To foster open-research, we chose to use the open-source physics engine PyBullet. The implementation chosen for this package allows to define very easily new tasks or new robots. This paper also presents a baseline of results obtained with state-of-the-art model-free off-policy algorithms. panda-gym is open-source and freely available at https://github.com/qgallouedec/panda-gym.
1 Introduction
Robotic reinforcement learning still struggles with sparse rewards and high interaction requirements, making simulation important for learning complex tasks. The paper proposes a simulated Panda environment while allowing new robots to be defined for its tasks.
1 Introduction
- Sparse rewards remain difficult for current reinforcement-learning algorithms, which may require many environment interactions to learn satisfactory models.Source: §1 · p. 1
- Learning complex tasks with sparse rewards currently requires simulation, and the Panda arm is widely used in robotics research.Source: §1 · p. 1
- The proposed Panda simulation also allows new robots to be defined for use with the available tasks.Source: §1 · p. 1
2 Environments
The environments provide goal-conditioned Panda tasks in PyBullet, with task-dependent observations, actions, and reward variants. The suite ranges from reaching to stacking, with task completion based on a 5 cm distance tolerance.
2 Environments
- The environments simulate a 7-degree-of-freedom Panda with a parallel gripper in open-source PyBullet and follow the Multi-Goal RL framework, sampling a new goal each episode.The observation includes desired and achieved goal vectors.Source: §2 · pp. 1–2
2.1 Tasks
- Five tasks progress from reaching and pushing to sliding, pick-and-place, and ordered two-cube stacking; completion requires coming within 5 cm of the target.Source: §2.1 · p. 2
2.2 Observation and action space
- Observations include gripper position and speed, object state when relevant, and gripper opening when unconstrained; actions command 3D motion and, where available, finger movement.Source: §2.2 · p. 2
2.3 Reward
- Sparse reward is the default: 0 within 5 cm of the goal and −1 otherwise; dense variants use negative distance, though designing dense rewards can require preference weights.Source: §2.3 · p. 3
3 Design Decisions
The design separates robot and task components so either can be extended independently. Its environment class routes actions to the robot and combines robot- and task-specific observations; tests report faster simulation than MuJoCo equivalents.
3 Design Decisions
- Separating robot and task classes lets developers define a new task without changing the robot, or a new robot without changing the task.Source: §3 · p. 3, Figure 2 · p. 3
- RobotTaskEnv sends agent actions to the robot, concatenates robot- and task-specific observations, and derives desired and achieved goals from the task.Source: §3 · p. 3
- The environments are on average 9.2% faster than MuJoCo equivalents; the comparison measured simulation time for 10^5 timesteps on one CPU core.Source: §3 · p. 3
4 Experimental results
The experiments establish baselines for three off-policy algorithms with HER across five tasks. DDPG solves reach and push, partially solves slide and pick-and-place, but does not solve stacking within the reported training budget; clipped double-Q ablations improve results in multiple environments.
4 Experimental results
- The baselines compare DDPG, SAC, and TD3 with HER, using success rates over 80 test episodes and interaction counts independent of the number of workers.Learning used eight CPU cores; trajectories lasted 50 timesteps except stacking, which used 100.Source: §4 · p. 4
- 100% DDPG success is reached on reach after 10^4 and push after 3 × 10^4 timesteps; slide and pick-and-place reach about 50%, while stacking stays near zero after 1.6 × 10^6.Source: Figure 3 · p. 4, §4 · p. 4
- Removing clipped double-Q significantly increases TD3 and SAC results in multiple environments.Source: §4 · p. 4
5 Conclusion and future works
panda-gym is a free, open-source package with five robotic tasks; its architecture supports defining new tasks and robots, while the reported algorithms solve some tasks but leave others unsolved.
5 Conclusion and future works
- The package includes five tasks, and the tested state-of-the-art algorithms solve some while leaving others unsolved.The package is free and open-source.Source: §5 · p. 4
- The architecture is designed to make it easy to define new tasks and new robots.Source: §5 · p. 4
- Planned extensions include peg-in-hole insertion, cube flipping, joint-value control, and observations with camera, force, or tactile modalities.Source: §5 · p. 4
A Hyperparameters
The experiments use the hyperparameters listed in Table 1, chosen to match those submitted by Plappert et al. [2018].
A Hyperparameters
- The experimental hyperparameters were chosen identical to those submitted by Plappert et al. [2018].Table 1 is identified as the hyperparameter listing.Source: A Hyperparameters · p. 6, Table 1 · p. 6
B Full results and ablations study
The ablations test removing clipped double-Q from TD3 and SAC and removing HER. In these environments, removing clipped double-Q has no effect or increases results, while the figure reports median success rates and interquartile ranges across 21 seeds.
B Full results and ablations study
- The TD3 and SAC ablations replace their target values; after clipped double-Q is removed, the second target Q-network no longer exists.Source: B Full results and ablations study · p. 7
- HER ablation removes storing k additional replay transitions per stored transition, each using a goal achieved later in the episode.The original transition is stored with its initial goal.Source: B Full results and ablations study · p. 7
- Removing clipped double-Q from TD3 and SAC either has no effect or increases results in these environments.The authors suggest reward scattering may prevent value overestimation, but say this intuition requires future verification.Source: B Full results and ablations study · p. 7
- Figure 4 compares HER and clipped double-Q ablations using median success rates and interquartile ranges across 21 random seeds.Source: Figure 4 · p. 8
C Environments specifications
Table 2 is identified as listing components of the observation and action spaces.
C Environments specifications
- Table 2 lists components of the observation and action spaces.Source: Table 2 · p. 9
D Overview of the learned policies
Figure 5 shows end-of-training policies for reach, push, slide, and pick & place, with each line representing a task. The image spacing varies by task, from one to four timesteps between successive images.
D Overview of the learned policies
- Reach is shown with one timestep between successive images.It is the top task in the figure.Source: Figure 5 · p. 9
- Slide is shown with four timesteps between successive images.It is the third task from the top.Source: Figure 5 · p. 9
- Push and pick & place are each shown with two timesteps between successive images.Push is second and pick & place is fourth from the top.Source: Figure 5 · p. 9