Source-linked AI summary
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, Sergey Levine
TL;DR
RL agents typically require extensive task-specific data, motivating pre-training from successful trials across previously seen tasks. PARROT learns an invertible behavioral prior for action generation and uses it to bootstrap exploration, accelerating new robotic manipulation tasks and outperforming prior approaches.
Problem
RL agents often require large amounts of task-specific data because they begin without prior knowledge and explore many unproductive behaviors.
Method
PARROT learns an invertible, state-conditioned mapping from noise vectors to actions using successful state-action pairs from previously seen tasks, then uses it to generate exploratory actions for downstream RL.
Results
PARROT solves all evaluated tasks substantially faster and achieves higher final returns than other methods, including tasks involving unseen objects and high-dimensional image observations.
Takeaways & Limitations
Behavioral priors can bootstrap exploration for new robotic manipulation tasks while retaining full control over the action space through the invertible mapping.
Takeaways & Limitations
The setup assumes tasks share fixed state and action spaces and are drawn from a task distribution, constraining the scope of the formulation.
Abstract
from arXiv · showhide
Reinforcement learning provides a general framework for flexible decision making and control, but requires extensive data collection for each new task that an agent needs to learn. In other machine learning fields, such as natural language processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged as a powerful paradigm to reduce data requirements when learning a new task. In this paper, we ask the following question: how can we enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors. We demonstrate the effectiveness of our approach in challenging robotic manipulation domains involving image observations and sparse reward functions, where our method outperforms prior works by a substantial margin.
1 INTRODUCTION
PARROT addresses the high data demands of RL by learning a behavioral prior from successful trials across tasks. The prior accelerates learning in new robotic manipulation tasks while preserving the agent’s ability to explore novel behaviors.
- RL agents often waste data on largely unproductive behaviors before discovering high-reward outcomes for each new task.Humans instead draw on prior experience to narrow the search space.
- The paper proposes learning an invertible mapping from noise vectors to actions, trained by maximizing the likelihood of actions observed in successful prior trials.Conditioned on observations, the mapping generates actions for a new MDP.
- Because the mapping is invertible, the RL agent retains access to every environment action and can still try behaviors distinct from those previously observed.
- PARROT pre-trains behavioral priors from diverse multi-task data and uses them to bootstrap exploration for new tasks.The framework targets robotic manipulation with image observations and sparse rewards.
- PARROT quickly learns tasks involving previously unseen objects in settings where RL from scratch fails to learn a policy.The paper also reports substantial improvement over prior methods using prior data.
2 RELATED WORK
The related work positions PARROT among demonstration-based, generative, hierarchical, and meta-learning approaches. Its distinction is using data from many prior tasks to accelerate RL for a new task rather than relying on new-task demonstrations or active interaction with prior tasks.
- Combining RL with demonstrations: Demonstrations can accelerate RL, but prior methods commonly combine imitation and RL for the same task.
- Combining RL with demonstrations: PARROT instead uses data from a wide range of prior tasks to speed RL for a new task.
- Generative modeling and RL: Unlike adversarial imitation methods, PARROT learns multimodal action distributions from data without environment interaction.
- Hierarchical learning: PARROT can be interpreted hierarchically, with a data-trained behavioral prior as the low-level policy and an RL-trained policy controlling it.
- Meta-learning: Meta-RL and meta-imitation methods also seek to accelerate learning on new tasks using experience from previously seen tasks.
3 PROBLEM SETUP
The setup assumes prior data contains behaviors structurally relevant to new tasks drawn from the same task distribution. PARROT learns from state-action data without reward labels, differing from meta-RL’s need for active interaction with prior tasks and rewards.
- Tasks are modeled as MDPs with fixed state and action spaces, while task dynamics and rewards may vary across a distribution p(M).The experiments use high-dimensional images as states.
- The behavioral prior is assumed to be trained on data containing behaviors that structurally resemble potential optimal policies for the new task.For example, prior object-picking behavior may help with a new object-placement task.
- Prior data is described as state-action trajectories from near-optimal policies for tasks drawn from p(M), while the new task is likewise drawn from p(M).
- The method requires only state-action tuples and does not require access to the reward function that would define trajectory optimality.The paper notes that every trajectory can be viewed as optimal for some unknown reward function.
- Unlike meta-imitation learning, PARROT can learn new tasks without demonstrations and does not require task-labeled, optimal, paired demonstrations.
4 BEHAVIORAL PRIORS FOR REINFORCEMENT LEARNING
PARROT learns a state-conditioned, invertible behavioral prior from successful state-action data, transforming noise into useful actions for downstream RL. The mapping improves exploration while preserving the agent’s control over the original action space.
- PARROT learns a state-conditioned mapping that transforms noise vectors into actions likely to be useful in the current state.The mapping is trained from near-optimal state-action pairs collected across previously seen tasks.
- The behavioral prior is trained with state-conditioned generative modeling and can reparameterize downstream RL policies through a = fφ(z; s).The task-specific policy controls the mapping’s noise input rather than directly selecting environment actions.
- PARROT uses an image-conditioned real NVP with convolutional image encoding to train one behavioral prior across visually observed tasks.The model uses four coupling layers conditioned on RGB observations.
- An invertible mapping can represent complex, multi-modal action distributions while allowing the RL agent to retain full control of the original action space.The prior increases the likelihood of useful actions without removing any reachable action.
- The mapping is learned once from prior data and then reused to accelerate learning on new tasks.Training the prior and the task-specific RL policy is decoupled, allowing different RL algorithms and generative models to be combined.
5 EXPERIMENTS
PARROT is evaluated on sparse-reward, image-based robotic manipulation tasks using diverse prior-task trajectories. It learns faster and reaches higher final returns than the compared methods, while performance depends on how prior behaviors match downstream actions.
- Domains: The evaluation uses 6-DoF robotic-arm manipulation with 7D actions, 48×48 RGB observations, randomized object positions, and sparse rewards.Test scenes contain novel objects absent from the training dataset, requiring visual inference of object positions.
- Experimental setup: The behavioral-prior dataset contains diverse trajectories whose repositioning behaviors structurally resemble behaviors needed by the new tasks.The experiments compare PARROT with SAC, BC-SAC, VAE-features, TrajRL, HIRL, and a prior-exploration ablation.
- Main results: PARROT learns substantially faster and achieves substantially higher final returns than other methods across the evaluated tasks.SAC and VAE-features fail to make progress, while TrajRL and HIRL improve but plateau earlier than PARROT.
- Dataset size: 10K, 25K, and 50K training trajectories yield similar performance, while even 5K trajectories outperform learning from scratch.The results indicate diminishing returns from collecting additional prior data beyond roughly 10K trajectories.
- Train-test mismatch: A prior trained only on pick-and-place tasks solves downstream grasping tasks well, but a grasping-only prior performs poorly on pick-and-place tasks.The authors associate this asymmetry with the unseen gripper-opening action required by pick-and-place tasks.
6 CONCLUSION
PARROT accelerates reinforcement learning on new tasks, including manipulation of unseen objects from high-dimensional images, but still requires thousands of trials for high success rates.
- PARROT learns behavioral priors from successful trials across a wide range of tasks.
- Learning from these priors accelerates reinforcement learning on new tasks, including unseen-object manipulation from high-dimensional image observations.
- PARROT compares favorably with other methods that use prior data to bootstrap learning on new tasks.
- Thousands of trials are still required to attain high success rates.
Appendices
The appendix algorithm trains a behavioral prior from previous state-action data, then uses a task-specific latent policy and environment feedback to learn the new task.
- The algorithm takes a dataset of state-action pairs from previous tasks and a new task M⋆ as input.
- It learns fφ by maximizing the likelihood term in Equation 2 before task-specific reinforcement learning begins.
- At each step, the latent policy samples z conditioned on the current state, and fφ maps z and the state to an environment action.
- The agent executes the action, observes the next state and reward, and updates the latent policy with the resulting transition.
B IMPLEMENTATION DETAILS AND HYPERPARAMETER TUNING
The implementation uses conditional normalizing-flow and convolutional architectures alongside several comparison methods and a shared SAC configuration.
- Behavioral prior: The behavioral prior is a conditional real NVP with four affine coupling layers, trained with Adam at a 1e−4 learning rate for 500K steps.
- Behavioral prior: A coupling layer transforms input noise into z′ using learned scale and translation functions conditioned on image features.
- Baselines: HIRL uses a conditional variational autoencoder with image conditioning and models individual actions for closed-loop control.
- Baselines: Behavior cloning and SAC share the Figure 9 policy architecture, while SAC uses the same hyperparameters across all reinforcement-learning experiments.
C.1 TASKS
The evaluation includes eight robotic manipulation tasks, with four shown in Figure 3 and four additional tasks shown in Figure 10.
- Four evaluation tasks are depicted in Figure 10, complementing the four tasks shown in Figure 3.
- Figure 10 includes grasp-and-lift tasks involving a can, vase, and baseball cap.
- Its final row shows picking up the baseball cap and placing it on a marble cube.
C.2 DATA COLLECTION
The dataset was collected with scripted grasping and pick-and-place policies that operate for fixed horizons and add Gaussian action noise. These procedures generate demonstrations through explicit distance, gripper, lifting, and placement conditions.
- Data collection used scripted grasping and pick-and-place policies described in Algorithms 2 and 3.The policies were used to collect the dataset.
- Scripted Grasping: The scripted grasping policy moves the end effector toward an object, closes the gripper, lifts the object, and then stops.It uses a 0.02 distance threshold and runs for 25 timesteps.
- Scripted Grasping: The grasping policy adds Gaussian noise sampled as N(0, 0.1) to each action before stepping the environment.
- Both scripted procedures use a 0.02 threshold and a 25-timestep horizon.
- Scripted Pick and Place: The scripted pick-and-place policy grasps an object, moves toward a point above a container, opens the gripper, and marks placement as attempted.It invokes the scripted grasping procedure when the object is not grasped.
C.3 SIMULATION OBJECTS
The simulations use 3D object models from ShapeNet and PyBullet. The study distinguishes objects used for training from those used for testing.
- Simulation environments used 3D object models from the ShapeNet dataset and PyBullet object libraries.
- Figure 11 presents the objects used for training.
- Figure 12 presents the objects used for testing.