PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

Huy Ha, Pete Florence, Shuran Song

arXiv:2307.14535v2cs.RO

TL;DR

Robot learning needs scalable ways to acquire robust, reusable manipulation skills and effective ways to learn from the resulting data. The paper combines LLM-guided planning, sampling-based robot utilities, automatic success verification and retrying, and language-conditioned diffusion-policy distillation. The distilled policy improved absolute success rates by 33.2% on average across five domains while learning robust retrying behavior.

  • Problem

    Robot learning seeks scalable acquisition of robust, reusable manipulation skills across diverse tasks and effective learning from the collected data.

  • Method

    The framework uses an LLM to plan and ground tasks in 6DoF robot utilities, infer success conditions for verification and retrying, and distills successful experience into a multi-task language-conditioned visuomotor policy.

  • Results

    33.2% average absolute success-rate improvement was reported across five domains, with the distilled policy learning robust retrying behavior from its data-collection policy.

  • Takeaways & Limitations

    The integrated framework automatically generates, labels, and distills diverse robot-complete exploration experience into a multi-task visuo-linguo-motor policy.

  • Takeaways & Limitations

    The data-generation implementation is limited to simulation environments, requiring sim2real transfer; the evaluation only tests learning the root task, not all labeled subtasks.

Abstract

from arXiv · show

We present a framework for robot skill acquisition, which 1) efficiently scale up data generation of language-labelled robot data and 2) effectively distills this data down into a robust multi-task language-conditioned visuo-motor policy. For (1), we use a large language model (LLM) to guide high-level planning, and sampling-based robot planners (e.g. motion or grasp samplers) for generating diverse and rich manipulation trajectories. To robustify this data-collection process, the LLM also infers a code-snippet for the success condition of each task, simultaneously enabling the data-collection process to detect failure and retry as well as the automatic labeling of trajectories with success/failure. For (2), we extend the diffusion policy single-task behavior-cloning approach to multi-task settings with language conditioning. Finally, we propose a new multi-task benchmark with 18 tasks across five domains to test long-horizon behavior, common-sense reasoning, tool-use, and intuitive physics. We find that our distilled policy successfully learned the robust retrying behavior in its data collection procedure, while improving absolute success rates by 33.2% on average across five domains. Code, data, and additional qualitative results are available on https://www.cs.columbia.edu/~huy/scalingup/.

1 Introduction

The paper addresses scalable robot-data acquisition and effective learning from that data by combining language guidance with robot planners and a language-conditioned visuomotor policy. It introduces a benchmark spanning 18 tasks across five domains.

  • The framework targets both scalable manipulation-data collection and robust learning from collected data without expert demonstrations or engineered rewards.
  • An LLM recursively decomposes task descriptions into hierarchical plans, grounds them in 6DoF robot utilities, verifies success, and retries failures.The utilities generate diverse trajectories, while inferred success functions automatically label outcomes.
  • The distilled policy extends diffusion policy to multi-task language-conditioned closed-loop control from visual observations and natural-language task descriptions.
  • 33.2% average absolute success-rate improvement was reported across five domains, and the policy learned robust retrying behavior from data collection.
  • The benchmark contains 18 tasks across five domains requiring approximately 800 control cycles, common sense, tool-use, and intuitive physics.

2 Related Works

The related work situates the framework among efforts to scale robot-complete data and learn multi-task visuomotor policies. Its distinction is using an LLM as a data-collection planner before distilling successful trajectories into an observable-information policy.

  • Prior robot-learning work scales language and action labels through human annotation, visual-language models, procedural simulation labels, autonomous policies, and human videos.
  • Unlike LLM-as-final-policy approaches, this framework uses an LLM-based planner for suboptimal data collection and distills successful trajectories into an observable-information policy.
  • The paper extends diffusion-based policies into language-conditioned visuomotor learning for multi-task control.

3 Approach

The framework autonomously generates language-labelled robot experience by combining LLM task planning, 6 DoF exploration primitives, and verify-and-retry execution. It then distills these experiences into a language-conditioned multi-task visuomotor policy using an extension of diffusion policy.

  • Approach: The framework automatically collects task-directed robot experience from task descriptions without human demonstrations or manually specified rewards.It combines efficient exploration with multi-task visuomotor policy learning in a unified framework.
  • Language-guided data generation: An LLM recursively decomposes tasks into hierarchical task trees and grounds each plan into sequences of 6 DoF exploration primitive API calls.The primitives include sampling-based motion planning, geometry-based grasp and placement sampling, and articulated-manipulation motion primitives.
  • Language-guided data generation: Pseudo-random grasp, placement, and motion sampling generates diverse trajectories for rich six-degree-of-freedom manipulation.Sampling-based utilities can produce varied grasps, placements, and connecting motion trajectories.
  • Language-guided data generation: Trajectory segments receive dense language labels from both subtask and root-task descriptions during pre-order task-tree execution.This supplies language-labelled experience for downstream policy learning.
  • Verify and retry: A success-function code snippet inferred by the LLM verifies trajectories, labels them with success or failure, and enables retries after failed execution.Retries reuse the robot-utility sequence with a different random seed without resetting the simulation state.
  • Policy distillation: The distilled policy extends diffusion policy to multi-task language-conditioned control from task descriptions, proprioception history, and visual observations.It uses wrist-mounted and global views and outputs sequences of end-effector control commands; with a 10 dimensional action space, it runs at ≈35Hz on an NVIDIA RTX3080.

4 Evaluation

The evaluation tests task-directed exploration and distillation into a robust, generalizable visuo-linguo-motor policy across an 18-task, five-domain benchmark. Results show that 6DoF exploration, verify-and-retry collection, diffusion-based distillation, and larger language models support broader or more robust performance, while simulation-based generation leaves transfer and subtask-learning boundaries.

  • Data collection: 6DoF utilities provide non-zero exploration success across all catapult bins, unlike deterministic 2D LLM-as-Policy, producing better data for distillation.The reported average success rates are 16.3%, 3.3%, and 2.2% across the three bins.
  • Data collection: Verify-and-retry improves data-generation performance across domains, while removing it yields 0.0% success in the mailbox domain.The reported improvements are 2×, 3×, 8×, and 13× in transport, catapult, balance, and drawer domains.
  • Language models: The 13B Llama model outperforms the 7B model by +20.0% in planning accuracy and +38.3% in success-verification accuracy, although both struggle in complex planning domains.In the drawer domain, both models fail to account for opening and closing the drawer.
  • Distilled policy: The distilled policy inherits retrying behavior and improves average success rates, whereas the no-retry policy is brittle and performs 34.8% lower across domains.The paper reports improvements of +23.4% for no-retry and +33.2% for its policy in Table 2.
  • Distilled policy: Diverse diffusion-generated retry attempts continue improving balance-task success with additional time, while baselines plateau after their first attempts.The paper attributes high performance to combining verify-and-retry with diffusion-based generation of diverse retry actions.
  • Sim2Real transfer: A policy trained on domain-randomized synthetic data achieves 76% success in a real-world transport task involving five novel objects.The result averages ten episodes per object.
  • Limitations: The data-generation implementation is limited to simulation and requires sim2real transfer; the study evaluates learning only on root tasks, not all labeled subtasks.The authors leave learning reusable sub-skills and compositional generalization for future work.

5 Conclusion

The paper integrates LLM planning, sampling-based robot utilities, and policy learning to generate, label, and distill diverse manipulation experience. Its pipeline includes recursive task decomposition, state-aware planning, success-condition inference, and language-conditioned policy distillation.

  • Policy Distillation: The policy-learning component extends diffusion policies to language-conditioned visuomotor learning for multi-task behavior.
  • The framework automatically generates, labels, and distills diverse robot-complete exploration experience into a multi-task visuo-linguo-motor policy.
  • LLM Pipeline: The approach uses modular LLM prompts for specialized reasoning skills, making unexpected task-tree outputs traceable to individual modules.
  • LLM Pipeline: The LLM planner recursively decomposes tasks into subtasks while propagating the environment state to child tasks.
  • LLM Pipeline: For every task in the inferred task tree, the LLM generates a code-snippet success condition using available state APIs.

C.1 Data Generation

Data generation uses a Verify & Retry process and continues until each task has at least 500 successful trajectories. Although long-horizon tasks can be costly, collection occurs only once.

  • Each task receives at least 500 successful trajectories through the 6DoF Exploration Primitives with Verify & Retry.The mailbox domain required 2 days on 256 CPU cores because of long horizons and low success rates.

C.2 Network Architecture & Hyperparameters

The policy uses shared architecture and hyperparameters across domains, combining language and multi-view visual encoders. Evaluation also tests generalization to novel toy instances.

  • Network Architecture: Task descriptions use CLIP B/32 text features projected to 512 dimensions, while each camera view has a separate ResNet18-based vision encoder.The visual features are flattened, concatenated, and projected into a 512-dimensional vector.
  • Generalization to Novel Objects: The distilled policy is trained on 22 toys and tested on 8 novel toys to evaluate generalization across object shapes, colors, sizes, and geometry.Objects are rendered from a fixed camera in the figure.

C.3 Training

Each domain has a separately trained multi-task policy using the same architecture and hyperparameters, with training performed on a single NVIDIA A6000.

  • Networks are trained for 2 days on one NVIDIA A6000, although performance typically saturates after about 1 day.The reported result uses the best checkpoint.

D Utilities Implementation

The motion-planning implementation uses RRT with grasped-object-aware collision checking, while grasp and placement sampling relies on geometric simulator data.

  • RRT motion planning incorporates grasped-object-aware collision checking for dynamic grasping constraints.
  • The geometry-based grasp and placement sampler uses point clouds derived from depth maps, camera matrices, and simulator segmentation maps.

E Benchmark

The benchmark evaluates tabletop manipulation across multiple task domains using randomized simulation environments, with tasks targeting transport, drawers, tool-use, and intuitive physics.

  • The benchmark uses Mujoco tabletop manipulation with a UR5e, WSG50 gripper, Finray fingers, two cameras, and a 4Hz policy control rate.
  • Mailbox: The mailbox task requires closing the mailbox with a package inside and raising its flag within 200 seconds, or 800 control cycles.
  • Transport: The transport task requires placing a randomly posed toy into the left bin within 100 seconds, including generalization to unseen object instances.
  • Drawer: The drawer domain contains 12 object-to-drawer tasks, each requiring the specified object to enter the specified drawer within 120 seconds.
  • Catapult: The catapult domain contains three bin-targeting tasks that require moving a block onto the catapult arm and pressing a button to launch it.
  • Intuitive Physics: The intuitive-physics task requires balancing a randomly oriented bus on a block within 100 seconds, testing precision and failure recovery.

F Full Results

Results identify task-specific difficulty factors in the drawer domain, including object size, top-drawer reach limits, and unsuitable planar action primitives.

  • The reported drawer results therefore distinguish difficulty arising from object geometry, drawer reachability, and action-primitive design.
  • Larger objects, especially the vitamin bottle and horse toy, are more challenging because upright orientations do not fit inside the drawer.Effective behavior requires sideways grasps, while the small crayon box has the highest success rates among data-collection policies.
  • The top drawer is difficult because operation approaches the robot’s kinematic reach range, making small action or grasp errors costly.Collisions can cause the grasped object to drop or the drawer to close.
  • Planar action primitives can fail when a top-down drawer-handle grasp collides with the drawer body.In LLM-as-Policy (2D), this produces an invalid goal configuration for the motion planner.

G Real World Evaluation

Real-world evaluation transfers a separately trained policy to a UR5e setup matching simulation and tests it on five unseen objects placed at random poses.

  • A separate policy is trained on domain-randomized scenes for real-world transfer.The randomized factors include lighting, texture, and camera pose.
  • The real-world evaluation matches the simulation setup with a UR5e, WSG50 gripper, and Toyota Research Institute Finray fingers.
  • Five unseen objects with varied shape, size, and visual appearance are each evaluated for 10 episodes from random poses in the right bin.
Loading 2307.14535v2…