PaperlayerView exact revision on arXiv

Source-linked AI summary

GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs

Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, Lirui Wang

arXiv:2410.03645v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Robot simulation data and evaluation are costly to acquire at scale, and scalable policies must also address sim-to-real gaps. GenSim2 generates articulated and long-horizon tasks and demonstrations with multimodal models and planners, then trains a multi-task policy. It reports a 20% policy-performance improvement over training exclusively on limited teleoperation data, while noting remaining limitations in task generation and transfer scope.

  • Problem

    Robot interaction data and evaluation are expensive to acquire at scale, while data generation and policy learning must scale and minimize sim-to-real gaps.

    Source: §1 · pp. 1–2

  • Method

    GenSim2 uses multimodal and reasoning models to generate articulated, long-horizon tasks and demonstrations, including keypoint-based planning, and trains a multi-task point-cloud and language-conditioned policy.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    20% improvement in policy performance compared to training exclusively on limited teleoperation data, alongside generation of over 100 articulated-object tasks using 35 objects (over 200 instances).

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors present massive simulated data generation with multimodal LLMs as a promising way to reduce data-collection effort and solve real-world problems.

    Source: §1 · p. 3

    Foundation-model hallucinations and some remaining human involvement constrain task generation; zero-shot sim-to-real transfer is limited to 6-DOF tasks with limited point-cloud observations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Robotic simulation today remains challenging to scale up due to the human efforts required to create diverse simulation tasks and scenes. Simulation-trained policies also face scalability issues as many sim-to-real methods focus on a single task. To address these challenges, this work proposes GenSim2, a scalable framework that leverages coding LLMs with multi-modal and reasoning capabilities for complex and realistic simulation task creation, including long-horizon tasks with articulated objects. To automatically generate demonstration data for these tasks at scale, we propose planning and RL solvers that generalize within object categories. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. To utilize such data, we propose an effective multi-task language-conditioned policy architecture, dubbed proprioceptive point-cloud transformer (PPT), that learns from the generated demonstrations and exhibits strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the policy architecture, we show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data, which enhances the policy performance by 20% compared with training exclusively on limited real data.

1 Introduction

GenSim2 targets scalable, realistic simulation data and policies while reducing human effort and sim-to-real gaps. It combines task and demonstration generation with a multi-task policy, and reports improved real-world policy performance when simulated data is used with limited teleoperation data.

1 Introduction

  • GenSim2 uses multimodal LLMs to iteratively generate and verify articulated, long-horizon tasks and demonstrations, with a keypoint-based motion planner that generalizes across object categories and initial conditions.The pipeline is reported to scale to 100 tasks for multi-task policy training.Source: §1 · p. 2
  • The proposed proprioceptive point-cloud transformer (PPT) uses point-cloud observations and language conditioning for multi-task learning designed for sim-to-real transfer.The paper presents the combined pipeline and policy as a way to reduce data-collection effort and solve real-world problems using simulated data.Source: §1 · pp. 2–3
  • Over 100 articulated-object manipulation tasks were generated using 35 objects (over 200 instances), including 50 long-horizon tasks.The task-generation pipeline achieves 25% better success rates than previous works.Source: §1 · p. 2
  • 20% improvement in policy performance compared to training exclusively on limited teleoperation data.The approach reports promising results across 8 real-world articulated-object tasks through zero-shot transfer and co-training with real-world data.Source: §1 · p. 3

2 Related Work

Related work covers simulation task generation, multi-task policies using 3D information, and sim-to-real transfer. GenSim2 is positioned as combining multimodal task generation for complex manipulation with point-cloud policies and real-world transfer.

Task Generation in Robotic Simulation

  • Robotics simulators support benchmarks with tens to hundreds of hand-designed tasks, while asset, scene, task design, solvability checks, and transfer verification require substantial human effort.Prior approaches include domain randomization [34] [7] [35] [36], procedural assets [8] [37], and text-to-3D synthesis [38] [39] [40] [41]; GenSim [9], RoboGen [10], and RoboCasa [11] generate task suites or simulations in different settings.Source: §2 · p. 3

Multi-Task Policy Learning with 3D Information

  • Prior multi-task policies use language conditioning and 3D inputs such as RGB-D, voxels, point clouds, or keypoints; this work focuses on point-cloud transformers for articulated-object manipulation.The related methods include [46] [48] [49] [50].Source: §2 · p. 3

Sim-to-Real Transfer

  • Sim-to-real approaches include RL-based controllers, domain randomization, domain adaptation, and policy composition; the paper frames its approach as distilling foundation-model knowledge into policies with a potentially smaller semantic gap than rule-based simulation data.The cited approaches include [51] [52] [54] [34] [56] and foundation models [14, 13].Source: §2 · p. 3

3 Generating Tasks and Training Policies at Scale

GenSim2 scales task proposals, solver generation, multi-task policy training, and transfer evaluation. Its pipeline composes LLM-generated tasks, multimodal solver generation and planning, and a policy that combines point clouds, proprioception, and language.

3 Generating Tasks and Training Policies at Scale

  • The framework extends task complexity beyond top-down pick-and-place through grounded multimodal designs, generic 6-DOF motion planners and RL solvers, and scalable policy architectures.Its overview lists task proposal, solver creation, multi-task training, and generalization evaluation and sim-to-real transfer.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 Task Proposal

  • Primitive tasks are proposed by prompting an LLM to choose assets and generate a task description, assets, and executable task code from fixed asset and small hand-designed task libraries.Long-horizon tasks combine primitive tasks involving articulated and rigid-body objects, using top-down decomposition or bottom-up composition from a task library; reasoning LLMs improve task proposal performance in this stage.Source: §3.1 · p. 4

3.2 Demonstration Generation

  • For demonstration generation, multimodal LLMs produce planner configurations from rendered scenes and object keypoints, while kPAM is extended to generate object-centric end-effector actuation motions.The pipeline can refine solver outputs through reject sampling, and a complementary RL-based policy learner is also proposed.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Policy Learning and Tranfer

  • PPT separately tokenizes point clouds, proprioceptive states, and language, fuses them through transformer blocks, and predicts action sequences using a policy head.The implementation tests MLP, transformer-decoder, and diffusion heads; simulated point clouds are augmented during training, while real observations are sampled and cleaned for inference.Source: §3.3 · pp. 5–6, Figure 4 · p. 6

4 Experiment: Task and Data Generation

The experiments assess task-generation feasibility using execution and solution rates, then test solver components and compare GenSim2 with RoboGen. The reported findings favor multimodal and reasoning inputs, iterative refinement, chained prompts, and GenSim2's task-generation methods.

Evaluation Criteria

  • Task-generation evaluation uses execution rate for completing the pipeline without errors and solution rate for solving the generated task.The experiment section evaluates the feasibility of task generation and the effectiveness of the task and data pipeline.Source: §4 · p. 6

Ablation Study

  • Without visual input, task-generation success rates fall because actuation-pose constraints require object-structure details; reasoning capability also improves vanilla LLM solver-generation scores.The text links visual input to scene appearance and spatial relationships, and hypothesizes that reasoning traces reduce hallucinations in kPAM constraints.Source: §4 · pp. 6–7

Comparison with Baseline

  • GenSim2 exceeds RoboGen in primitive and long-horizon settings; bottom-up outperforms top-down, and OpenAI o1 improves task decomposition across both evaluation criteria.The comparison attributes the baseline advantage to fine-grained motion-planner generation and the bottom-up advantage to composing existing tasks rather than generating each subtask from scratch.Source: Table 1 · p. 7, §4 · p. 7

Ablation Study

  • Multi-shot rejection sampling outperforms the version without it, and more refinement iterations improve pipeline performance.The authors report that MLLM rejection sampling can produce irrelevant responses because of domain gaps in 3D robotic-scene perception.Source: §4 · p. 7
  • A prompt chain that generates constraints before actuation motions yields over 30% better results than directly generating the complete solver.The chain supports focused sequential processing, using earlier-stage outputs in later stages.Source: §4 · p. 7

5 Experiment: Multi-Task Policy Training and Transfer

The section evaluates multi-task policies trained on generated simulation data, their generalization to unseen instances, and their real-world transfer. It reports improved real-world policy performance when GenSim2 data is co-trained with real demonstrations.

5.1 Multi-task Training in Simulation

  • As training tasks increase, policy performance first drops and then rises; these tests use 10 demonstrations per task and 20 evaluation episodes per task.The policy is evaluated on four original tasks in a low-data regime.Source: Figure 6 · p. 8, §5.1 · p. 8
  • PPT success rates drop by less than 3% on unseen object instances, while RGB-input policies reduce by considerably more.The study attributes generalization to object- and spatial-level variance in generated data and the point-cloud policy design.Source: §5.1 · p. 8

5.2 Real-World Experiments

  • GenSim2 simulation data enables zero-shot sim-to-real transfer, with the resulting policy outperforming one trained solely on limited real-world data.The real-world evaluation uses 100 simulation demonstrations per task and 10 additional teleoperated demonstrations.Source: §5.2 · p. 8, Table 2 · p. 9
  • 20% in absolute values and 50% in relative scale: co-training GenSim2 data with real-world demonstrations enhances policy performance.The evaluation covers 8 real-world tasks and compares simulation-only, real-world-only, and combined training setups.Source: §5.2 · p. 8, Table 2 · p. 9

6 Limitations and Discussions

The authors identify limitations in task generation and in the scope of their zero-shot sim-to-real evaluation.

6 Limitations and Discussions

  • Foundation-model hallucinations and some remaining human involvement constrain complex task generation; zero-shot transfer is evaluated only on 6-DoF tasks with limited point-cloud observations.The authors cite a lack of robotic-centric knowledge, including 3D spatial understanding, as a source of hallucinations.Source: §6 · p. 9
  • Task generation still requires human involvement, although the authors describe it as minimal.This is a stated boundary on the pipeline's reduction of human effort.Source: §6 · p. 9
  • Foundation models such as GPT-4V can hallucinate when creating meaningful tasks and coding them, given limited robotic-centric knowledge.The passage names 3D spatial understanding as an example of the missing knowledge.Source: §6 · p. 9

7 Conclusion and Future Work

GenSim2 combines task and demonstration generation using multimodal foundation models with a point-cloud policy intended to transfer generated demonstrations to the real world.

7 Conclusion and Future Work

  • GenSim2 uses multimodal foundation models to generate up to 100 robotic simulation tasks, including long-horizon manipulation with articulated objects.The conclusion also describes a simulated demonstration-generation pipeline and a multi-task point-cloud policy for transfer to the real world.Source: §7 · p. 9
  • Future work includes increasing task complexity and diversity through advanced multimodal agents and 3D asset generation, and exploring sim-to-real methods for multiple embodiments.Source: §7 · p. 9

A.1 Asset Statistics

  • Each articulated asset retains one unfixed prismatic or revolute joint; other joints are fixed, and rigid-body assets adapt YCB objects from ManiSkill2.Object scales and joint limits were modified to make scenes suitable.Source: §A.1 · p. 15

A.2 Task Generation Statistics

GenSim2 generated 100 tasks split evenly between primitive and long-horizon tasks, while task counts treated instances of the same object class as one task.

A.2 Task Generation Statistics

  • 100 generated tasks comprised 50 primitive tasks and 50 long-horizon tasks.Source: §A.2 · p. 15
  • Tasks using different instances of the same object class were counted as one task.Source: §A.2 · p. 15
  • The generated tasks did not cover every asset, and the asset library could be expanded with more object classes and instances.Source: §A.2 · p. 15

A.3 Task Solver Details

The section describes complementary task solvers, human-effort measurements, demonstration-generation evaluation, task scaling, PPT implementation, and benchmark comparisons.

A.3 Task Solver Details

  • kPAM builds object-centric actuation trajectories from keypoint constraints, while an LLM-rewarded PPO learner handles tasks too ambiguous for planner constraints.kPAM is prioritized for smoother, generalizable motions and takes about 2 seconds per configuration; RL is used for tasks it cannot solve.Source: §A.3 · p. 15

A.4 Human Verification

  • Task creation time fell by 50%, with over 90% of the required time spent waiting for the LLM.Keypoint labeling averaged 8.2 seconds per task, and GenSim2 task creation averaged about 4 minutes across five tasks.Source: §A.4 · p. 16, Table 3 · p. 17

A.5 Demonstration Generation Statistics

  • Demonstration-generation success was measured taskwise over 24 single-step and 15 long-horizon tasks, using five runs of 50 randomized episodes each.Primitive tasks used position and yaw randomization; some hard long-horizon tasks used half-scale noise when normal randomization prevented successful demonstrations.Source: §A.5 · p. 17, Table 5 · p. 19

B.1 Task Settings

  • The task list scales from four tasks to 24 by progressively adding task groups.The 24-task set includes door and refrigerator tasks added after the 20-task set.Source: §B.1 · p. 20

B.2 Architecture Implementation

  • PPT combines a transformer policy stem and action head with encoders for language, point clouds, and proprioception.The task tokenizer is frozen, PointNext is fine-tuned, and proprioception is encoded by an MLP trained from scratch.Source: §B.2 · p. 20

B.3 Policy Architecture Comparison against Baselines.

  • On RLBench, PPT had the best performance on 6/10 tasks, improving over competitive baselines PerAct [46] and GNFactor [70], with reduced camera-view and pretrained-feature requirements.Source: §B.3 · p. 20

B.4 Additional Experiments for Object-Level Generalization

The policy showed a small success-rate drop on unseen simulated instances and achieved reported generalization across real-world tasks and objects.

B.4 Additional Experiments for Object-Level Generalization

  • Success rates dropped by around 3% on unseen object instances.Training used data from 90% of instances, with the remaining 10% held out; tests used training sets of 5 and 15 tasks with randomized initial poses.Source: §B.4 · pp. 20–21, Figure 10 · p. 21

C Real-World Details

  • The real-world evaluation selected eight articulated tasks, and policy inference took 0.1s on an NVIDIA 3080 GPU.The evaluation included laptop, safe, drawer, bucket, and box manipulation tasks.Source: C Real-World Details · p. 21

C.1 Generalization on Unseen Objects

  • On multiple real-world tasks, success exceeded 60% with unseen objects, and performance across objects within a task was comparable.Source: Table 7 · p. 22, §C.1 · p. 22

C.2 Co-Training Experiment Details

The co-training setup paired teleoperated real-world demonstrations with simulated demonstrations for each generated task.

C.2 Co-Training Experiment Details

  • The study collected 10 teleoperated demonstrations for each generated task.Source: §C.2 · p. 22
  • Each task's co-training data combined 100 episodes of simulation data with 10 demonstrations of real data.Source: §C.2 · p. 22
  • An HTC Vive tracked 6-DOF hand movements, mapped to robot-arm pose control through a position-velocity controller.Source: §C.2 · p. 22

D Comparison with Previous Works

This section contrasts GenSim2’s 6-DOF task and demonstration-generation focus with prior frameworks and reports task-generation and transfer comparisons.

D Comparison with Previous Works

  • Table 8 compares features of generated data across released robotics-simulation generation frameworks.Source: Table 8 · p. 23, D Comparison with Previous Works · p. 23
  • 57.5% success rate for transfer via co-training with both simulation and real data, versus RoboCasa’s score lower than 25%; zero-shot sim-to-real transfer reaches 42.5%.Source: D Comparison with Previous Works · p. 23
  • GenSim2 focuses on realistic 6-DOF tasks, unlike GenSim’s top-down pick-and-place, RoboGen’s dynamic tasks, and RoboCasa’s mobile manipulation tasks.GenSim2’s demonstration-generation method differs from RoboGen’s RL-based approach, which the authors describe as producing unrealistic, untransferable data; RoboCasa requires substantial human teleoperation for source data.Source: D Comparison with Previous Works · pp. 22–23
  • 60% improvement in task-generation success rates, with multimodal task-solver generation outperforming previous work by 20%.Source: D Comparison with Previous Works · p. 23

E Prompt Examples

Appendix E presents prompt examples for generating and solving robotic simulation tasks, including structured task and solver configurations and reasoning-LLM configuration for closing a safe door. The examples also illustrate limitations and failures across task proposal, decomposition, solver creation, and motion planning.

E.1 Prompt Templates

  • The task-generation prompt constrains assets, physics, feasibility, and single-motion goals, then requests a validated task with a structured Python-dictionary format.It asks the LLM to avoid bad or repeated examples and use specified success criteria.Source: §E.1 · p. 24
  • Long-horizon task decomposition is limited to at most five single-motion subtasks, each with one asset, with grasp and ungrasp represented separately.Other subtasks use a prescribed dictionary structure and success-criteria vocabulary.Source: §E.1 · pp. 24–25
  • Code-generation prompts provide reference code and require matching its structure, using known functions and assets, and commenting on the generated code.A follow-up prompt asks for the task code in a Python code block.Source: §E.1 · p. 25
  • The kPAM prompts define solver configs through contact and axis constraints plus pre- and post-actuation motions, with varied motion trials expressed relative to the actuation pose and manipulator base.The templates describe point-to-point and axis-parallel or orthogonal constraints, including tolerances and target inner products.Source: §E.1 · pp. 25–27

E.2 Example for Multi-modal LLM Response

  • The close-drawer multi-modal example shows a solver-config request with scene and frame-axis images, followed by contact and orientation constraints and pre-actuation motions.The accompanying vanilla-LLM example is explicitly reported to fail to generate correct constraints.Source: §E.2 · pp. 28–30

E.3 Example for Reasoning LLM Response

  • The close-safe reasoning-LLM example explains keypoints and feasible contact and axis constraints, while the paper highlights current vision-language model limitations and anticipates that progress such as API fine-tuning could address failure modes.The passage says the example is compared with a vanilla LLM, but does not provide the comparison outcome in this excerpt.Source: §E.3 · p. 31
  • The safe-closing example specifies handle contact and aligns the gripper’s Z-axis with the door’s closing direction and its X-axis with the hinge axis.The configuration encodes contact and frame-axis constraints for the actuation pose.Source: §E.3 · p. 32
  • The example requests three diverse pre-actuation motions for the same task and shows a translate-z motion to lift the gripper above the handle.The motions use a mode-and-value format, with translation distances in meters or rotation angles in radians.Source: §E.3 · p. 33
  • For post-actuation, the example reasons through pushing the door forward and outputs a move-forward plan within the motion and step limits.It also notes that plans should follow the robot’s coordinate system and avoid table collisions.Source: §E.3 · pp. 33–34
  • A separate vanilla-LLM example is marked as failing to generate correct constraints for the safe-door task.The shown configuration includes gripper-to-handle contact and axis-alignment constraints.Source: §E.3 · p. 35

E.4 Example Failure Cases

  • The task-proposal example fails by proposing an unlock-safe task that assumes the safe handle can rotate, although the handle is not rotatable.Source: §E.4 · p. 36
  • The tidy-up-foam-brick decomposition fails because it grasps the object before opening the microwave door, reversing the required order.Source: §E.4 · pp. 36–37
  • The microwave solver example fails because its generated actuation pose cannot complete the task.The example’s solver configuration includes contact and frame-axis constraints for the gripper.Source: §E.4 · p. 38

Related papers

Loading 2410.03645v1…