Source-linked AI summary
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, Sergey Levine
TL;DR
The paper asks how robots can learn useful, transferable skills from offline data without manually specified rewards or online exploration. It trains goal-conditioned Q-functions with hindsight relabeling and offline-specific techniques, finding diverse real-robot skills, long-horizon goal chaining, and downstream benefits. The method remains limited by its need for a suitable goal image consistent with the current scene.
Problem
Robotic skill learning from offline data lacks manually specified rewards and additional online exploration, limiting how existing experience can be reused for general-purpose learning.
Method
The method learns goal-conditioned Q-functions from offline data using hindsight relabeling, synthetic negative labels, and goal chaining across trajectories.
Results
The method learns diverse skills on real robots from high-dimensional goal images, reaches long-horizon goals through goal chaining, and improves downstream task learning through pre-training or auxiliary objectives.
Takeaways & Limitations
Offline goal-reaching models can provide reusable robotic representations and support zero-shot, jointly trained, or fine-tuned downstream behaviors.
Takeaways & Limitations
At test time, specifying a task requires a suitable goal image consistent with the current scene, limiting goals from different scenes or object types.
Abstract
from arXiv · showhide
We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data. In particular, we propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We employ goal-conditioned Q-learning with hindsight relabeling and develop several techniques that enable training in a particularly challenging offline setting. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects. We also show that our method can learn to reach long-horizon goals across multiple episodes through goal chaining, and learn rich representations that can help with downstream tasks through pre-training or auxiliary objectives. The videos of our experiments can be found at https://actionable-models.github.io
1. Introduction
The paper targets general-purpose robotic skill learning from offline data without hand-specified rewards or online interaction. It proposes goal-conditioned policies that treat dataset states as goals, enabling diverse skills, long-horizon goal chaining, and downstream task use.
- General-purpose robots need broad skill repertoires, but learning each skill separately requires manually programmed rewards and newly collected data.
- The paper studies learning entirely from offline data without hand-specified rewards or online interaction, using previously attempted tasks as reusable experience.
- Goal-conditioned policies learn to reach any possible goal state from any current state, turning every dataset state into a potential task.
- Actionable Models address offline goal-conditioned learning with synthetic negative labels and goal chaining for temporally extended skills.
- The learned models support zero-shot visually indicated tasks, auxiliary-loss joint training, and fine-tuning with task-specific rewards.
2. Related Work
Related work spans goal-conditioned learning, reward-free supervised approaches, offline Q-learning, and explicit dynamics models. The paper distinguishes its approach by using offline model-free Q-learning to stitch trajectories without demonstrations, online collection, or an additional cost function.
- Goal-conditioned RL commonly uses hindsight relabeling to reuse trajectories for reaching many goals.
- Goal-conditioned supervised methods seek useful control information without rewards, but prior approaches use demonstrations or online data collection.
- The paper uses dynamic programming through Q-learning to stitch multiple trajectories and reach goals without observing a complete trajectory from the starting state.
- The method adapts conservative offline Q-learning and QT-Opt to goal-conditioned control for large datasets and complex real-world tasks.
- Unlike explicit dynamics models that predict future observations, goal-conditioned Q-functions estimate whether a particular goal will be reached and directly provide an optimal action without a cost function.
3. Preliminaries
The preliminaries formulate reinforcement learning over an MDP and replace manually defined task rewards with sparse goal-specific rewards. Temporal-difference learning yields a goal-conditioned Q-function, whose greedy actions define the reaching policy.
- An MDP is specified by state and action spaces, transitions, rewards, an initial-state distribution, a discount factor, and a task horizon.
- RL learns a policy maximizing expected discounted trajectory reward under the environment transitions and policy-induced actions.
- Each goal uses a sparse reward of 1 when the terminal state equals that goal and 0 otherwise.
- The goal space consists of goal images, and episodes terminate upon reaching the specified goal, making the maximum total reward 1.
- Temporal-difference learning produces a goal-conditioned Q-function, and the policy acts greedily by selecting the action with maximum Q-value.
4. Actionable Models
Actionable Models learns goal-conditioned skills from offline trajectories by combining hindsight relabeling with conservative Q-learning, then extends the method to chain goals across trajectories. The resulting model outputs a goal-conditioned policy and supports value propagation across multiple episodes.
- 4. Actionable Models: Actionable Models learns a goal-conditioned Q-function from offline trajectories, using each trajectory subsequence as evidence for reaching its final state.The policy acts greedily with respect to the learned Q-function: π(a|s,g) = arg max_a Q(s,a,g).
- 4.1. Goal-conditioned offline Q-learning: Hindsight relabeling supplies positive reaching examples, but unseen actions lack evidence and can receive overestimated Q-values without regularization.The method therefore treats unseen actions conservatively rather than relying on relabeling alone.
- 4.1. Goal-conditioned offline Q-learning: The conservative objective assumes unseen deviations fail to reach the requested goal unless dataset trajectories demonstrate recovery, minimizing Q-values for sampled negative actions.Negative actions are sampled near the Q-function decision boundary using a soft-max distribution over their Q-values.
- 4.2. Goal chaining: Goal chaining extends learning beyond single trajectories by assigning the final transition either a success reward or its Q-value for the external goal.This lets value evidence propagate from a later trajectory through a chaining point into an earlier trajectory.
- 4.2. Goal chaining: The chaining procedure can connect more than two episodes, with its effective horizon limited by Bellman value propagation and discounting.The method samples goals from the dataset and can bias sampling toward goals with high Q-values.
- 4.3. Method Summary: The complete method minimizes a loss whose first term raises Q-values for reachable goals and whose second term regularizes the Q-function.The TD target uses reward 1 when the next state equals the goal and otherwise bootstraps from the discounted policy value.
5. Applications of Actionable Models
Actionable Models learn goal-reaching skills from offline datasets and support downstream reinforcement-learning tasks through pre-training, auxiliary objectives, or task-specific fine-tuning. The framework combines trajectory relabeling, goal chaining, and conservative Q-target computation within replay-based Q-learning.
- Applications of Actionable Models: The framework learns goal-reaching skills from previously collected offline datasets and integrates with Q-learning methods using experience replay.Algorithm 1 provides example extraction and Q-target computation components for this integration.
- Applications of Actionable Models: Relabeled trajectory subsequences assign reward 1 to their final states and add the resulting transitions to the replay buffer.Subsequences are produced by randomly cutting sampled trajectories before relabeling.
- Applications of Actionable Models: Random dataset goals receive the final state's Q-value as a relabeled reward, enabling goal chaining across trajectories.The relabeled trajectories use R(τ0:i, grand) = Qθ(si, ai, grand) for sampled goals.
- Applications of Actionable Models: Q-target computation applies the Bellman equation and samples negative actions weighted by exp(Qθ), filtering actions too close to the observed action.Actions are first sampled uniformly from the action space and then reweighted by their Q-values.
- Applications of Actionable Models: Pre-trained goal-conditioned Q-functions can be fine-tuned on task rewards, while auxiliary goal-reaching losses can be optimized alongside conventional online RL.The auxiliary objective mixes relabeled and original trajectories with probability ξ and conditions original trajectories on an all-zero goal image.
6. Experiments
Experiments evaluate Actionable Models for offline goal-conditioned learning in simulation and on real robots, including visual generalization, long-horizon goal chaining, and downstream representation learning. The method learns diverse skills from heterogeneous offline data and outperforms or improves on the reported baselines across these settings.
- 6.3. Simulated visual goal reaching experiments: The experiments test offline goal-conditioned policies against behavioral cloning and Q-learning baselines in simulated manipulation tasks.Evaluation uses pick-and-place, stacking, fixture placing, and food-object grasping tasks, with success rates reported from online trials that are excluded from training.
- 6.3. Simulated visual goal reaching experiments: Actionable Models outperform goal-conditioned behavioral cloning on all simulated tasks, while unregularized Q-learning fails and random-goal negatives learn only the simplest task.The authors attribute the baseline failures to the lack of useful negative action examples in offline goal-conditioned training.
- 6.4. Real-world visual goal reaching experiments: The real-robot model learns goal-reaching skills from heterogeneous data across high-dimensional camera observations and is evaluated on grasping, placing, and other object-manipulation tasks.Training combines data from 12 conventionally rewarded tasks, with source-data success rates ranging from 5% to 55%, and uses a QT-Opt architecture augmented with a goal image.
- 6.4. Real-world visual goal reaching experiments: The learned policy transfers manipulation skills to objects absent from training, including silverware, a towel, and a rubber cord.Goal images are recorded in the same scene used during execution before objects are reset to random initial positions.
- 6.5. Goal chaining: Goal chaining enables reaching long-horizon fixture-placing goals across separate grasping and placing episodes, whereas disabling it causes a substantial performance drop.In the reported ablation, success decreases from 66% to 8% on longer-horizon container-placing tasks when goal chaining is disabled.
- 6.6. Representation learning: Goal-conditioned pre-training and a 10% auxiliary objective mix-in significantly accelerate downstream instance-grasping training, including in real-robot fine-tuning.Without pre-training, real-world standard offline QT-Opt achieves less than 5% success; fine-tuning a pre-trained model reaches at least 20% on all three tasks using fewer than 10K episodes.
7. Discussion and Future Work
The method learns diverse goal-reaching skills from high-dimensional offline robot data and supports downstream learning through pre-training or auxiliary objectives. Its main boundaries are dependence on suitable test-time goal images and difficulty with extended-horizon tasks involving many object relocations.
- The method learns diverse skills on real robots from high-dimensional goal images in a completely offline setting.
- Goal-conditioned Q-functions are regularized to avoid overestimating unseen actions, while goal chaining enables reaching goals across multiple episodes.
- Downstream reinforcement-learning tasks can be learned more efficiently by fine-tuning pre-trained goal-conditioned models or adding a goal-reaching auxiliary objective.
- At test time, specifying a task requires a suitable goal image consistent with the current scene, limiting general task descriptions such as object-category commands.
- The approach currently cannot reposition a large number of objects in a single episode when goals require reasoning over extended horizons.
- Training goal-conditioned Q-functions from diverse multi-task offline data can provide broad world knowledge for direct goal reaching or faster downstream policy training.