PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomously Acquiring Robot Manipulation Skills with Language-Driven Quality-Diversity

Émiland Garrabé, Mahdi Khoramshahi, Stéphane Doncieux

arXiv:2608.30983v1cs.RO

TL;DR

Robot QD skill acquisition must balance autonomous learning with the adaptability provided by diverse motion-primitive archives. The paper uses LLMs to explore fitness and behavior-descriptor function spaces and adapts multi-BD MES, generating diverse archives across four manipulation tasks while requiring only free-form task descriptions. The approach still depends on a fixed motion template and needs broader template and sim2real development for wider task coverage.

  • Problem

    Autonomous skill-learning methods typically produce single solutions, while QD methods require experts to design success, fitness, and diversity functions.

  • Method

    The method samples low-dimensional fitness and behavior-descriptor function spaces with LLMs and uses the resulting heterogeneous descriptors in multi-BD MES.

  • Results

    Across 4 robotic manipulation tasks, the method generates diverse motion-primitive archives and outperforms inferred and hand-written QD parametrizations.

  • Takeaways & Limitations

    LLM-based exploration can autonomously parameterize QD skill acquisition from free-form task descriptions while producing archives of diverse motion primitives.

  • Takeaways & Limitations

    The paper identifies task-tailored motion templates and domain-randomization adaptation as needed to broaden tasks and improve sim2real performance.

Abstract

from arXiv · show

Quality-diversity (QD) algorithms have been gaining traction in robot learning, where diverse motion primitive libraries allow robots to adapt zero-shot to constraints at deployment time. However, such methods typically require expert designers to write the success condition, fitness and diversity metrics, and this strongly limits the robot's autonomy. On the other hand, existing LLM-based reward-shaping techniques allow robots to learn autonomously but only output single high-performing solutions, limiting the robot's adaptability. In this paper, we propose an approach designed to output diverse motion primitive archives by autonomously leveraging quality-diversity algorithms, only requiring a free-form description of the task in common language. To address the difficulty of designing relevant fitness and diversity metrics, we propose an autonomous exploration mechanism able to reliably output sets of functionals covering the fitness and behavior descriptor (BD) space. First, we pose policy exploration as a functional design problem, where the functional spaces are lower-dimensional than the full BD and fitness spaces, and propose an LLM-based exploration scheme to sample from these low-dimensional spaces without any task-specific prompts, fine-tuning or expert intervention. We adapt a multi-BD variant of the MAP-Elites success (MES) algorithm, designed to leverage the heterogeneous BD samples. Finally, through experiments based on the genesis simulator, we show that our method effectively generates archives of diverse motion primitives, outperforming classical QD algorithms with inferred and hand-written parametrizations on a set of $4$ robotic manipulation tasks.

I. INTRODUCTION

Robot skill acquisition faces a trade-off between autonomous LLM-based learning that typically yields single solutions and QD methods that produce diverse archives but require expert-designed signals. The paper adapts LLM reward shaping to QD so diverse motion primitives can be acquired from language instructions.

  • Motivation: QD archives provide diverse motion primitives that can improve robustness to unseen task requirements.QD methods seek diversity across candidate solutions, supporting exploration of non-convex fitness landscapes.
  • Research gap: Standard QD methods require experts to curate success conditions, fitness signals, and diversity metrics for each new skill.Hand-designed functions are typically developed through trial and error, while unsupervised diversity representations can reduce explainability.
  • Related approaches: LLM reward-shaping pipelines infer reward code from open instructions, bridging open-semantics task formulations and learning algorithms.Related approaches also refine reward code iteratively through feedback or trajectory ranking.
  • Motivation: Most autonomous skill-acquisition methods focus on single high-performing solutions, limiting adaptability in high-dimensional manipulation spaces.The paper links this brittleness to deployment in the real world and carefully curated curricula.
  • Contribution: The paper combines LLM-based reward design with QD to generate diverse solution archives while targeting autonomous manipulation-skill acquisition.This addresses the contrast between autonomous single-solution learning and expert-dependent diverse-solution methods.

III. METHOD

The paper formulates archive design as selecting motion primitives likely to solve unknown instances of a known language-specified task. It explores fitness and behavior-descriptor function spaces with LLM sampling and uses the resulting candidates in multi-BD MES.

  • 1) Design problem:: The design problem seeks an archive most likely to achieve unknown task instances given a known task.Task instances add requirements from designer needs or deployment-time obstacles.
  • 1) Design problem:: The environment, robot, and object are represented as a POMDP, although most POMDP assumptions are not strictly necessary.This framework provides a general formalization for the task and archive-design problem.
  • 1) Design problem:: Unlike single-primitive reward design, the proposed formulation targets a set of motion primitives that can solve varied instances of the same task.The formulation assumes a dedicated selection and planning process can exploit the archive.
  • QD formulation: QD methods measure behavioral diversity using distances in behavior space or separate occupied niches in a discretized archive.The paper uses a variant of MAP-Elites Success with a success condition to assess individuals.
  • Method overview: The method replaces expert input with sampling strategies intended to cover fitness and behavior-descriptor spaces.The generated samples parameterize the QD search rather than relying on manually curated functions.

C. LLM inference for multi-BD MES

The method explores fitness and behavior-descriptor functionals through simpler, separately sampled spaces, then combines heterogeneous descriptors in a multi-BD MES algorithm. One success condition is inferred because it is considered easier to design and computationally cheaper to keep fixed.

  • Success condition: The method infers one success condition per task because success-criterion design is considered relatively easy.Using one condition reduces computational load while fitness and BD spaces are explored more broadly.
  • Fitness exploration: Fitness is treated as a time-step average, reducing the need to search the intractably large fitness space.MES fitness is especially important during early discovery, then becomes less influential among successful elites.
  • Behavior descriptor: Behavior-descriptor exploration factorizes candidate functions into information extraction and measurement-timestep choices.The sampled BD space is {g ◦h : g : I →R∞; h : I^N →I}, whose component spaces are simpler to explore.
  • Behavior descriptor: Candidate information functions and timesteps are sampled without replacement to avoid duplicates and improve coverage.The two spaces are explored through separate inferences to manage context size, then their candidates are composed.
  • Multi-BD MES: Multi-BD MES concatenates individuals selected from each grid before mutation and evaluation, followed by local competition in every BD grid.This contrasts with standard MES, which selects individuals from the archive and applies local competition through a single archive process.

IV. EXPERIMENTS

Experiments implement the autonomous QD pipeline in Genesis using a fixed motion-primitive template, Algorithm 1, and repeated task trials. The setup infers functions and constructs archives through multi-BD MES.

  • Experimental setting: The experiments validate the method in Genesis, using GPU parallelization to evaluate individuals rapidly.Each method is attempted five times per task, with metrics reported as mean ± std.
  • Implementation: Candidate solutions use a 6-dof initial pose and a 3-dof Cartesian motion vector with constant gripper orientation.The motion vector is constrained to 5–20 cm, with positive vertical motion and initial rotations sampled from a downward-pointing half-sphere.
  • Algorithm 1: Algorithm 1 infers a success condition, fitness functions, timestamps, and measured quantities from a task description and simulation API.Fitness and BD-related candidates are sampled without replacement, then measured quantities and timestamps form a BD mesh.
  • Algorithm 1: The pipeline evaluates each inferred fitness function with multi-BD MES before selecting the archive size-maximizing fitness for final archive generation.The final archive is generated using the selected fitness and the same BD mesh.

B. Implementation: perception API

The perception API provides simulator information about object states, contacts, gripper pose, and articulated-object configurations. Experiments cover grasping, drawer, faucet, and push environments.

  • Perception API: The LLM receives object pose, table and object contacts, gripper pose, and articulated-object state information from simulation.The API includes Euler-angle orientations and joint positions or displacements for articulated objects.
  • Tasks: The evaluation includes four Genesis manipulation tasks: grasping, drawer, faucet, and push.Example environments are shown in reading order for these four tasks.
  • Task conditions: The grasping task requires the object to touch the gripper without touching the table.Its ground-truth behavior descriptor is the gripper xyz position at first object contact.
  • Task conditions: The faucet task requires rotating the hot-water tap by at least 0.314 radians.Its ground-truth behavior descriptor is the tap opening at episode end.

3) Open the drawer at least 10cm:

The drawer and push-related evaluation uses expert-defined task conditions and ground-truth behavior descriptors, while comparing compute-matched QD and genetic baselines. Archive size and ground-truth descriptor coverage measure successful and diverse solutions.

  • Drawer task: The drawer must extend its prismatic joint at least 10cm past the closed position.The drawer task’s ground-truth behavior descriptor is the initial gripper xyz position.
  • Push task: The push task requires placing the mug on the table at least 5cm from its starting position.Its ground-truth behavior descriptor is the object xy position at episode end.
  • Baselines: Baselines are compute-matched using the same population size and evaluation count.They include naive LLM inference, hand-crafted MES, and a generic genetic algorithm.
  • Metrics: Success archive size counts successful individuals, while ground-truth BD coverage measures diversity in task approaches.Coverage projects each output archive into the hand-crafted ground-truth BD space.

V. RESULTS AND DISCUSSION

Across four simulated manipulation tasks, the method generates larger and more diverse motion-primitive archives than the compared approaches. Its heterogeneous diversity streams are especially useful for hard exploration, while longer runs may benefit hand-crafted descriptors.

  • Discussion: Longer experiments may allow hand-crafted MES to eventually outperform inferred baselines in ground-truth BD coverage.The paper attributes its current smaller archives partly to sparse describing BDs and uses 50 generations per experiment.
  • Grasping task: More than twice as many grasps are found as with naive inference on the hard grasping task.The method also strongly outperforms hand-crafted MES.
  • Drawer task: The drawer results outperform baselines in ground-truth BD exploration and overall archive size.This extends the reported advantage beyond the grasping task.
  • Results: Across four tasks, the method outperforms baselines while requiring substantially less parametrization effort than standard QD algorithms.The approach autonomously explores BD and fitness spaces and uses a multi-BD MES variant.
  • Faucet task: All methods fill the ground-truth BD space on the faucet task, but the proposed method finds significantly more trajectories.This task is characterized as an easy exploration problem.
  • Push task: The proposed method again finds significantly more push solutions, while the genetic algorithm has extremely poor ground-truth BD coverage.The ground-truth BD space is easily explored in this task.

APPENDIX I PROMPTS

The appendix prompt frames skill acquisition as language-driven quality-diversity learning in Genesis. It asks the LLM to identify task-relevant simulator quantities for fitness and behavior descriptors.

  • BD inference: The BD inference procedure separately samples timestamp functions and measured quantities before composing them into descriptors.The fitness prompt is similar but omitted in the appendix excerpt because of space constraints.
  • Prompt setup: The prompt targets simple manipulation skills learned as joint-space trajectories by a robot with a parallel gripper.It specifies an approximately 15cm gripper length from finger tips to center of mass.
  • Quality-diversity: Quality-diversity uses fitness to evaluate candidate solutions and behavior descriptors to encourage high-performing diversity.The intended outcome is an archive of diverse, high-performing solutions.
  • Prompt setup: The task is supplied as a free-form placeholder, “TASK,” within the prompt.The pipeline is intended to use common-language task descriptions.
  • Simulator interface: Each simulated candidate produces an infos list containing dictionaries of measured physical quantities.The LLM is asked to use these available simulator functions when selecting relevant information.

{PERCEPTION API}

The perception API guides LLM-based design of behavior descriptors by selecting simulation timestamps and extracting low-dimensional physical quantities from recorded information. These extracted quantities support later BD computation in the quality-diversity pipeline.

  • Behavior descriptor design: Behavior descriptors distinguish candidate solutions to promote different approaches and exploration in behavioral space.The descriptor measures diversity rather than candidate performance, which is evaluated separately through fitness.
  • Timestamp selection: Four relevant simulation timestamps are proposed, including fixed moments or events such as first gripper-object contact.The selected timestamps are used to compute the behavior descriptor.
  • Function implementation: Python extraction functions return one-dimensional tuples from the simulator information and return None when a requested timestamp is unavailable.These functions are later used to compute the behavior descriptor.
  • Simulator information: The API provides simulator quantities at an automatically selected timestep through an information dictionary.Each candidate is rendered in simulation, and relevant information is stored before the timestep is selected.
  • Quantity selection: LLM prompts request four simple physical quantities for the BD, generally as low-dimensional 2-to-5-value tuples.Positions and orientations are preferably retained as full 3D values rather than partially selected components.

APPENDIX II EXAMPLE INFERRED FUNCTIONS

The appendix illustrates inferred fitness and behavior-descriptor functions for the “Grasp the mug” task. The examples combine object, gripper, contact, and timing signals.

  • Fitness samples: The example fitness signals include object vertical position, negative gripper-object distance, and binary gripper-object contact.Fitness is averaged over simulation timesteps.
  • Behavior-descriptor samples: The example BD signals include first gripper-mug contact and the first timestep without mug-table contact.These event-based signals identify distinct interaction stages during the grasping task.
  • Behavior-descriptor samples: Additional BD information includes gripper roll-pitch-yaw orientation and a two-value contact state for gripper-mug and mug-floor contact.The contact state records whether each specified contact is present.
Loading 2608.30983v1…