GenSim: Generating Robotic Simulation Tasks via Large Language Models

Lirui Wang; Yiyang Ling; Zhecheng Yuan; Mohit Shridhar; Chen Bao; Yuzhe Qin; Bailin Wang; Huazhe Xu; Xiaolong Wang

Source-linked AI summary

GenSim: Generating Robotic Simulation Tasks via Large Language Models

Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, Xiaolong Wang

arXiv:2310.01361v2Oct 2, 2023cs.LGcs.CLcs.CVcs.RO

TL;DR

GenSim addresses the limited task-level diversity of simulation data by using LLMs to generate simulation environments, tasks, and demonstrations. Its goal-directed and exploratory generation modes expand task libraries for multitask policy training, which improves generalization in simulation and real-world transfer. The approach remains limited by coding errors, imperfect evaluation, manual filtering, and its focus on tabletop pick-and-place tasks.

Problem
Simulation data is cheaper than real-world interaction data, but existing generation methods provide limited task-level diversity because novel tasks require substantial human design and verification effort.
Method
GenSim uses LLM prompting, retrieval, task libraries, and code generation in goal-directed and exploratory modes to create simulation tasks and demonstrations for multitask policy training.
Results
Policies trained with GPT4-generated tasks improve task-level generalization, including 50% better generalization, around 40% zero-shot transfer in simulation, and 25% better real-world generalization.
Takeaways & Limitations
Scaling simulation task libraries with LLM-generated tasks supports stronger generalization to new tasks in simulation and the real world.
Takeaways & Limitations
Generated code can contain syntax errors, hallucinations, grounding failures, and evaluation misalignment, and the study explores only tabletop pick-and-place task generation.

Abstract

from arXiv · show

Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data to demonstrate significant task-level generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GenSim, has two modes: goal-directed generation, wherein a target task is given to the LLM and the LLM proposes a task curriculum to solve the target task, and exploratory generation, wherein the LLM bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, on which we conduct supervised finetuning and evaluate several LLMs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world and outperform baselines by 25%. See the project website (https://liruiw.github.io/gensim) for code, demos, and videos.

1 INTRODUCTION

GenSim addresses the difficulty of creating diverse, verifiable robotic tasks by using LLMs to generate simulation task code and demonstrations. The resulting task library expands benchmarks and supports multitask policy training for stronger task-level generalization.

Motivation: Real-world interaction data is expensive, while simulation has mainly improved scene- and instance-level diversity rather than task-level diversity.Creating new tasks requires specifying asset relationships and progression while checking achievability and transferability.
Approach: GenSim uses LLM grounding and coding to automatically design and validate task asset arrangements, task progression, and simulation tasks.The generated simulation data distills LLM reasoning and coding capabilities into language-conditioned visuomotor policies.
Approach: The framework combines prompting, a task library of high-quality generated instructions and code, and language-conditioned multitask policy training.It operates in goal-directed and exploratory modes for task generation.
Evaluation: Over 100 tasks are generated from 10 human-curated tasks, creating an expanded benchmark for evaluating task generation and policy learning.The study also evaluates several LLMs and applies supervised finetuning to improve task generation.
Results: 50% improved generalization and around 40% zero-shot transfer to new simulation tasks are reported for policies trained jointly with GPT4-generated tasks.These policies outperform training on only human-curated tasks in zero-shot generalization.
Results: 25% better real-world generalization is reported when policies are pretrained on diverse GPT4-generated simulation tasks.The result concerns sim-to-real transfer to unseen tasks.

2 AUTOMATED TASK GENERATION FOR POLICY LEARNING

GenSim generates simulation task programs through prompting, retrieval, and iterative use of a task library, then converts those tasks into demonstrations for multitask policy learning. Its task representations also support composition, retrieval, clustering, and policy training.

Task creator: Goal-directed and exploratory prompt chains generate task code, with the mode selected according to whether the target task is known.Goal-directed generation supports a specified task or curriculum, while exploratory generation proposes novel tasks.
Task creator: The task creator generates novel task descriptions and code implementations, which are decomposed into scene generation and demonstration generation.The framework uses the Ravens benchmark’s parameterized pushing and pick-and-place primitives.
Task library: Few-shot prompting retrieves reference tasks and code from the task library before generating new task descriptions and implementations.The references guide implementation of task classes, scene construction, assets, spatial goals, and language goals.
Policy learning: Generated task code produces scenes, simulations, and expert demonstrations that support imitation learning and policy training.GenSim adds execution feedback, LLM criticism, task-library feedback, and rollout checks to the generation pipeline.
Task composition: Color-ordered-insertion illustrates task-level composition by combining elements of block-insertion and place-block-in-bowl.Learning jointly with this related, more complex task is reported as potentially improving generalization to put-block-in-bowl.
Task library: The task library stores previously generated high-quality tasks for retrieval, validation, language-model finetuning, and multitask policy training.Exploratory outputs can bootstrap iterative training of task creators for goal-directed generation.
Task representation: Task code embeddings provide a representation for clustering and policy training that is less sensitive to perceptual variation than language instructions.The embedding space groups tasks by objects or motion types, including rope and structure-building tasks.

3 EXPERIMENTS

GenSim experiments evaluate LLM task generation, policy generalization, and sim-to-real transfer. Generated tasks improve coding evaluation, policy generalization, and robustness on long-horizon real-world tasks, while revealing coding and distribution failure modes.

The experiments ask whether LLMs can generate simulation tasks, whether generated tasks improve policy generalization, and whether they benefit real-world deployment.
LLM task generation: 120 diverse tasks were generated, and a two-stage prompt chain with few-shot examples and a task library improved code-generation success rates.
Task-level generalization: Training on more generated tasks produced stronger zero-shot generalization, with joint training improving original CLIPort-task performance by over 50% under a 5-demo regime.
LLM task generation: GPT4 outperformed other models on held-out simulation coding tasks, while finetuned open-source models approached state-of-the-art performance but sometimes misaligned objectives and implementations.
Task-level generalization: Diverse exploratory tasks were more beneficial for generalization across novel tasks and domains, whereas related compositional tasks supported transfer to related tasks.
Sim-to-real transfer: Pretraining on diverse simulation tasks improved robustness on long-horizon real-world tasks, including build-wheel, place-bread, and sort-garbage after adaptation.
Additional analysis: Policies trained on GPT4-generated tasks achieved average success rates of 75.8% single-task and 74.1% multi-task, close to human-curated-task rates of 76.6% and 76.1%.
Additional analysis: Generated tasks showed varied colors, assets, actions, and instance counts, including many scenes with over 7 object instances and pick-and-place actions.

4 RELATED WORK

Related work covers LLM reasoning and coding, simulation task and scene generation, and language models in robotics. GenSim is situated among methods using LLMs for task creation, verification, and robotic program generation.

Reasoning and Coding via LLMs: Prior work explored LLM reasoning, task exploration, environment feedback, self-verification, and code generation for novel tasks.
Task and Scene Generation for Simulation: Simulation research has addressed domain randomization, procedural asset generation, and text-to-3D methods for scene diversity.
Language Models in Robotics: Robotics applications of language models include policy learning, task and motion planning, log summarization, and synthesizing policy and optimization programs.

5 CONCLUSION AND FUTURE WORKS

GenSim uses LLMs to generate and organize diverse simulation tasks, supporting multitask policies that generalize across simulated and real-world tasks. The framework combines prompting, retrieval, finetuning, and task-code analysis, while identifying remaining generation and evaluation limitations.

Conclusion: GenSim augments robotic simulation with LLM-generated tasks and uses the resulting data to train policies that generalize to new simulated and real-world tasks.The framework investigates prompting, retrieval-augmented generation, and finetuning for task-code generation.
Framework: The task library supports task generation by retrieving prior descriptions and code, while also providing data for multitask policy training.It is initialized from human-curated benchmark tasks and expanded with generated tasks.
Limitations: The framework still produces syntax errors, hallucinations, and insufficient physical or geometric grounding, and its evaluation metric can require manual filtering.The authors also note that only tabletop tasks were explored.
Task analysis: GPT embeddings of generated task codes were projected with PCA and t-SNE, then clustered with KNN into six groups associated with objects and motions.The visualization indicates structure in the generated task-code space.
Task analysis: The 120 generated tasks showed varied colors, assets, actions, and instance counts, with multiple objects commonly appearing in scenes.Common colors included red, blue, yellow, and green.

A.4 TRAINING DETAILS

The finetuning experiments train LLMs to map task names and prompts to simulation code. Finetuning substantially improves Code Llama’s code structure and brings its performance closer to GPT-3.5.

Finetuning setup: Finetuning uses 112 generated tasks, with task names and short prompts as inputs for producing task code.The setup targets domain-specific simulation-task implementation and offers cheaper costs and faster inference than prompting.
Code Llama: Code Llama produced example or random answers under prompting, whereas finetuning yielded much more structured code.The prompting failure occurred in both few-shot and zero-shot settings.
Results: Finetuned Code Llama achieved performance similar to GPT-3.5, while broader self-bootstrapping and larger code-example training remained future work.

A.5 TASK CREATION LIMITATIONS

The task-generation benchmark faces recurring quality and scope limitations, including duplicated tasks, mismatched descriptions, ungrounded motions, noisy language, imbalanced coverage, and restricted task complexity.

Repeated Tasks: Different names and implementations can describe the same underlying task, reducing the effective diversity of the generated benchmark.Examples include multiple color-matching insertion and sorting tasks.
Mismatched Language Instructions: Generated language instructions can disagree with task implementations, and the benchmark metric may fail to detect this mismatch.One example describes placing each cylinder on its matching stand while the implementation stacks cylinders on one block.
Ungrounded Motion Sequence: Some generated motion sequences are not implemented, achievable, or well grounded in the benchmark’s physical setting.Examples include bowl stacking, lifting with a spatula, and pushing a block into other objects.
Noisy Descriptions: LLM-generated task names, descriptions, and instructions contain noisy or interchangeable terminology that can obscure task goals.Terms such as “rearrange,” “sort,” “insert,” and “place” are often used interchangeably.
Imbalanced Tasks: Bootstrapping biases generation toward common color-coordinated pick-and-place tasks, while the benchmark excludes overly complicated tasks and limits coverage to 100 tasks.Rope and pile tasks appear more rarely, and task definitions remain subjective.
Task Complexity: The work explores only top-down pick-and-place tasks, leaving more dexterous, higher-degree-of-freedom, and contact-rich tasks for future work.

B SIMULATION TRAINING EXPERIMENT

The simulation experiments use the CLIPort setup with a UR5e suction-gripper robot and top-down RGB-D observations. Multitask training must account for relations among tasks because semantically similar instructions can conflict, while related task selection improves robustness.

Simulation setup: The experiments use a UR5e with a suction gripper and top-down RGB-D observations from a RealSense camera.The setup follows the original CLIPort training configuration.
Training resources: Training 100 tasks can require up to 5 days on 8 GPUs, while multitask training uses 4 or 8 GPUs for 2 days.
Multitask training: Joint datasets can contain conflicting instructions when tasks share similar language but specify different object relations.The examples compare “put-block-in-bowl” with “place-red-in-green.”
Multitask training: Policies trained only on the 10 original CLIPort tasks achieved lower than 5% success on GPT-generated tasks.The authors use task relations and code embeddings to select related tasks for joint training.

C REAL WORLD EXPERIMENT

The real-world setup uses an overhead RGB-D camera with an XArm-7 and suction gripper. The policy is limited to generalizing across new tasks rather than compositional variation in colors or object instances.

Real-world setup: An XArm-7 with a suction gripper uses an overhead bird’s-eye-view camera to capture RGB-D observations.Workspace boundaries and background subtraction generate a depth-image mask.
Scope and limitation: The policy generalizes to new tasks rather than compositionally generalizing across different colors or object instances.The authors attribute the choice of pick-and-place tasks partly to the available suction-cup hardware.

D.1 TASK DESCRIPTION EXAMPLE

The task-description examples illustrate how GenSim proposes diverse tabletop manipulation tasks while constraining assets, physical feasibility, specificity, and novelty. Its prompt chain separates task description from implementation and uses reflection to filter proposals.

Task examples: GenSim examples combine colored objects, spatial arrangements, stacking, insertion, navigation, and precision challenges into tabletop tasks.Examples include multicolor bridges, matching balls to boxes, and arranging zones or blocks by color.
Task-description constraints: The prompt requires available assets, specific counts, shapes, colors, physical feasibility, and non-overlap with existing tasks.The task output includes a name, one-sentence description, and asset list.
Generation modes: Goal-directed generation asks for simpler tasks that contribute skills toward a target task, whereas bottom-up generation seeks diverse feasible tabletop tasks.Both modes first generate natural-language descriptions and then code implementations.
Prompt context: The task-creation prompt uses reference tasks, prior task names, bad examples, available assets, and additional rules as context.These inputs support few-shot generation while discouraging duplicated or physically invalid designs.
Reflection and filtering: GenSim prompts an LLM to accept or reject proposed tasks and retains a task only when three parallel critics agree.The critics use temperature 0.5 to obtain diverse judgments before requiring agreement.

D.3 EXAMPLE TASK LIBRARY CONVERSATIONS

Task-library conversations show GenSim using retrieved examples and critics to assess novelty and implementation correctness. Proposals can be rejected for minor overlap or for code that fails to realize the stated task.

Task-library retrieval: The task library supplies prior descriptions and code examples that guide reference-task selection for new proposals.One example retrieves four color-coordinated construction and insertion tasks as coding references.
Critic agreement: The example illustrates that agreement requires both meaningful task differences and code that matches the language description.One critic catches the coding error while another flags repetition with minor differences.
Novelty checking: A proposed rearrangement task is rejected because it closely resembles the existing color-ordered-insertion task and adds insufficient novelty.The critic treats the difference between matching and differently colored fixtures as minor.
Code validation: A second critic identifies that the implementation does not enforce the described left-to-right color order, despite the proposal’s additional manipulation challenge.The code randomly places fixtures and objects and adds goals according to the colors array.

D.4 EXAMPLE IMPLEMENTATION ERRORS

The implementation-error examples catalog common failures in generated simulation code and the checks used to filter them. Errors span assets, language goals, dimensions, poses, syntax, runtime execution, and demonstration generation.

Common implementation errors: Common generation failures include nonexistent assets, ambiguous goals, incorrect goal dimensions, tuple-vector misuse, oversized or invisible objects, and unfilled URDF templates.The examples also include incorrect end-effector construction and invalid tuple indexing or mutation.
Error book: The error book records recurring mistakes such as random target poses, too few language goals, and concrete Python exceptions.Examples include type, value, index, and array-indexing errors during environment reset or reward computation.
Validation pipeline: GenSim checks generated code progressively for syntax, runtime execution, and successful demonstration generation before retaining tasks.These checks detect missing paths, out-of-bounds behavior, and failures to produce demonstrations.
Human verification: Human verification averages around 10 seconds per task, with success rates above 50%.The table reports both human verification time and pass rates for generated tasks.

D.9 TOP-DOWN EVALUATION EXAMPLE

The top-down evaluation compares multiple LLM-generated implementations of the cylinder-in-colorful-container task. The examples vary from single-cylinder placement to multi-cylinder color matching and ordered insertion.

Zero-shot Code Llama: The zero-shot Code Llama implementation defines a single cylinder and a colorful container for insertion.Its task template describes putting the cylinder in the colorful container.
Few-shot Code Llama: The few-shot Code Llama implementation expands the task to color-specific cylinders and containers with a specified sequence.The implementation uses red, blue, green, and yellow and sets a 20-step limit.
GPT-3 and GPT-3.5: GPT-3 and GPT-3.5 implementations describe placing cylinders into containers of matching colors, with GPT-3.5 generating multiple colored cylinders and containers.The GPT-3 example uses a single-cylinder description, whereas the GPT-3.5 examples define color lists and corresponding containers.
GPT-4: The GPT-4 implementation specifies four distinct-colored cylinders and matching containers, with a four-step maximum and a same-color placement goal.The colors are red, blue, green, and yellow.