Source-linked AI summary
RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian Ge, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, Peize Sun, Haibao Yu, Chao Yang, Wenqi Shao, Wenhai Wang, Jifeng Dai, Yu Qiao, Mingyu Ding, Ping Luo
TL;DR
Robotic behavior synthesis still struggles to translate multimodal understanding into precise actions that generalize across scenarios and robot platforms. RoboCodeX uses tree-structured multimodal code generation, supported by specialized reasoning data and iterative self-updating fine-tuning, to produce physically grounded behaviors. It achieves state-of-the-art performance across simulated and real manipulation tasks and competitive navigation performance, including a 17% success rate improvement over GPT-4V.
Problem
Robotic behavior synthesis lacks a reliable way to translate high-level multimodal understanding into precise, generalizable control across tasks and robot platforms.
Method
RoboCodeX decomposes instructions into object-centric manipulation units with physical constraints and preferences, then generates robot-specific code using specialized multimodal reasoning data and iterative self-updating fine-tuning.
Results
RoboCodeX achieves state-of-the-art performance across four manipulation-task types in simulated and real robots, competitive navigation performance, and a 17% success rate improvement over GPT-4V.
Takeaways & Limitations
The framework provides a multimodal code interface that translates semantic and perceptual understanding into tailored robotic behaviors across manipulation settings.
Abstract
from arXiv · showhide
Robotic behavior synthesis, the problem of understanding multimodal inputs and generating precise physical control for robots, is an important part of Embodied AI. Despite successes in applying multimodal large language models for high-level understanding, it remains challenging to translate these conceptual understandings into detailed robotic actions while achieving generalization across various scenarios. In this paper, we propose a tree-structured multimodal code generation framework for generalized robotic behavior synthesis, termed RoboCodeX. RoboCodeX decomposes high-level human instructions into multiple object-centric manipulation units consisting of physical preferences such as affordance and safety constraints, and applies code generation to introduce generalization ability across various robotics platforms. To further enhance the capability to map conceptual and perceptual understanding into control commands, a specialized multimodal reasoning dataset is collected for pre-training and an iterative self-updating methodology is introduced for supervised fine-tuning. Extensive experiments demonstrate that RoboCodeX achieves state-of-the-art performance in both simulators and real robots on four different kinds of manipulation tasks and one navigation task.
1. Introduction
RoboCodeX addresses the gap between high-level multimodal understanding and precise, generalizable robotic control by translating instructions into object-centric, physically grounded code. It combines tree-of-thought decomposition, multimodal reasoning data, and iterative self-updating fine-tuning, achieving state-of-the-art performance across simulated and real robotic tasks.
- Motivation: RoboCodeX targets the challenge of mapping high-level semantic understanding to generic robotic behaviors across diverse objects, constraints, and robot platforms.Prior approaches struggle with novel scenarios, precise action sequences, and coordinating manipulation preferences across multiple objects.
- Approach: Code serves as a bridge from multimodal semantic understanding to low-level motions tailored to robot mechanics, enabling transfer across morphologically distinct robots.The representation is intended to connect MLLM cognition with precise robotic planning and control.
- Approach: RoboCodeX uses a multimodal tree-of-thought structure to decompose instructions into object-centric manipulation units with target poses, physical constraints, preferences, and trajectories.The framework combines visual, depth, and language inputs to produce feasible, safe, and optimized motion plans in cluttered environments.
- Training: RoboCodeX is trained with a specialized multimodal reasoning dataset and an iterative self-updating framework that generates high-quality executable code for supervised fine-tuning.The dataset uses simulated environments with diverse tasks, while self-updating improves training data and code success rates.
- Results: 17% success rate improvement over GPT-4V is reported, alongside state-of-the-art performance across four manipulation-task types in simulated and real robot systems.The contribution statement also reports competitive performance on embodied navigation tasks.
2. Related Works
Prior embodied-AI systems use language models, perception modules, or multimodal models for planning and understanding, but connecting perception and reasoning to precise grounded action remains difficult. RoboCodeX addresses this gap through visually grounded code generation that expands action nodes with object- and motion-specific information.
- Code Generation for Robotic Control: Earlier code-generation methods produce executable robot behaviors from linguistic inputs but lack multimodal grounding for adapting actions to perceived object characteristics.ProgPrompt, Code-as-Policies, and related approaches generate plans or policy code, while RoboCodeX incorporates visual observations into behavior synthesis.
- Embodied AI with Large Foundation Models: Embodied-AI systems commonly provide language models with textual scene descriptions, perception APIs, visual context, or predefined action primitives, but grounded action remains an open challenge.Perception, reasoning, and action are often handled through separate mechanisms rather than one integrated process.
- Embodied AI with Large Foundation Models: RoboCodeX bridges multimodal cognition and precise robotic planning through code generation that predicts target positions, physical properties, preferential rankings, and feasible trajectories.Each code node is expanded with multimodal predictions needed for grounded control.
- Multimodal Large Language Models: Although MLLMs have advanced visual instruction following and grounding, their multimodal reasoning for complex robotic behavior synthesis remains under-explored.Existing MLLM developments include visual question answering, instruction tuning, region description, and localization.
3. Methods
RoboCodeX decomposes long-horizon instructions into object-centric units, reasons over multimodal geometry and physical preferences, and compiles the resulting predictions into executable, dynamically feasible robot trajectories. Its methods combine multi-view 3D perception, tree-of-thought code generation, procedurally generated and iteratively refined training data, and hierarchical visual feature integration.
- 3.1. Problem Setup: RoboCodeX decomposes high-level instructions into object-centric units and generates trajectories that encode dense end-effector waypoints and discrete gripper actions.The formulation optimizes task completion and control costs subject to dynamics and kinematics constraints for each sub-task.
- 3.2. Multi-modal Tree-of-thought Code Generation: Three RGBD views are fused into a TSDF, while visual reasoning partitions tasks into sequential units with language descriptions and manipulation preferences.The preferences specify approach directions or contact positions, such as center grasps for spherical objects and edge grasps for open containers.
- 3.2. Multi-modal Tree-of-thought Code Generation: Each unit is expanded through part-level affordance prediction, grasp-pose proposals, physical-property prediction, plane detection, and trajectory planning with collision and singularity avoidance.AnyGrasp supplies the top 10 grasp candidates, visual reasoning selects among them, and GAMMA estimates articulated-object parts and joint parameters.
- 3.3. Dataset Preparation: The pre-training dataset procedurally combines HM3D scenes, sampled objects, GPT-4 task descriptions, and executable programs filtered for syntax and quality.General vision-language data are mixed with generated data to reduce overfitting.
- 3.3. Dataset Preparation: Supervised fine-tuning uses verified simulator and real-world examples, searches alternative manipulation settings for failed code, and adds GPT-4V coding annotations.Manually revised zero-success examples are incorporated into the human-labeled pool, while general vision-language data remain part of SFT.
- 3.4. Vision Language Model Design: The vision-language model compresses visual tokens with a Q-Former and aggregates hierarchical vision-transformer features through a channel-wise attention adapter.The compressed visual representations are concatenated with text tokens from code and documentation prompts to reduce sequence length and integrate multi-scale information.
4. Experiments
Across manipulation, navigation, real-world robot transfer, and ablations, RoboCodeX demonstrates broad task and platform generalization, with preference prediction, visual adaptation, and general VQA data contributing to performance.
- 4.1. Evaluation on Manipulation Task: RoboCodeX outperforms GPT-4V on pick-and-place, articulated-object, and long-term manipulation tasks, reducing errors linked to grasp-pose selection and trajectory planning.Its multimodal code-generation training reduces the error margin relative to GPT-4V, while remaining errors are primarily associated with grasp poses and trajectory planning.
- 4.2. Evaluation on Embodied Navigation Task: RoboCodeX achieves better embodied-navigation performance than selected foundation-model baselines and is comparable to GPT-3.5.Evaluation uses success rate and success weighted by path length on HM3D and HSSD through multimodal reasoning over observations and candidate frontiers.
- 4.3. Evaluation on General VQA: RoboCodeX shows general visual reasoning comparable to LLaVA-1.5-13B without specialized robotic-task fine-tuning.The evaluation uses LLaVA-Bench and MM-Vet to assess multimodal conversation and complex multimodal capabilities.
- 4.4. Real World Experiments: In zero-shot real-world tests, RoboCodeX adapts to Franka Emika and UR5 robots by changing only the robot configuration file.The tests include multi-stage pick-and-place and put-object-in-drawer tasks across different robot platforms and scenarios.
- 4.5. Ablation Study: Preference prediction significantly outperforms direct highest-scoring grasp selection across tasks, improving manipulation stability and alignment with subsequent planning.The ablation evaluates average success rate over four task types and 50 trials.
- 4.5. Ablation Study: Removing the vision adapter lowers success rates because fine-detail object understanding worsens, although the adapter is not the model’s most critical component.The results identify preference prediction as more central to overall performance than the vision adapter.
- 4.5. Ablation Study: Removing general VQA data causes significant overfitting, poorer instruction following across objects, improper answers, and a substantial drop in average success rate.The finding supports incorporating general VQA data during supervised fine-tuning to prevent overfitting.
5. Conclusion
RoboCodeX bridges multimodal language models and robotic control by translating semantic understanding into tailored behaviors. Experiments report state-of-the-art performance in complex manipulation tasks across simulated and real-world environments.
- 5. Conclusion: RoboCodeX bridges multimodal large language models and robotic control systems by translating semantic understanding into tailored robotic behaviors.
- 5. Conclusion: Its multimodal tree-of-thought approach, specialized dataset, and iterative fine-tuning methodology support interpretation of visual observations and human instructions into precise robot-specific actions.
- 5. Conclusion: RoboCodeX demonstrates state-of-the-art performance in complex robotic manipulation tasks in both simulated and real-world environments.
A. Manipulation Simulation Setup
The manipulation simulation uses a Franka arm in a Gazebo environment containing PartNet-Mobility cabinets and tabletop objects. Multi-view RGBD perception produces 3D object representations that feed grasp detection.
- A. Manipulation Simulation Setup: The simulation integrates a Franka robotic arm with PartNet-Mobility cabinets and tabletop objects in a dynamic environment.
- A. Manipulation Simulation Setup: Gazebo renders the environment and physical interactions, while ROS, MoveIt, and OMPL support robot operation, motion planning, and obstacle avoidance.
- A. Manipulation Simulation Setup: Multi-view 2D bounding boxes are aligned using depth-derived geometric correspondences before entering the 3D perception loop.
- A. Manipulation Simulation Setup: The perception loop combines RGBD images and 2D grounding to generate 3D bounding boxes, point clouds, and occupancy maps for objects.
- A. Manipulation Simulation Setup: Generated 3D point clouds are passed to a grasp detection module that produces grasp poses.
B. Implementation details of Vision Langauge Model
The vision-language model follows a BLIP-2-style architecture with EVA-CLIP visual features and a LLaMA-13B language model. Its vision adapter transforms intermediate vision-transformer features into q-former-compatible representations.
- B. Implementation details of Vision Langauge Model: RoboCodeX uses a BLIP-2-style architecture with EVA-CLIP ViT-G/14 visual features and a pretrained LLaMA-13B language model.
- B. Implementation details of Vision Langauge Model: The q-former and language model are fine-tuned using ShareGPT4-V, then initialized for multimodal code-generation pretraining and specialized fine-tuning.
- B. Implementation details of Vision Langauge Model: Frozen vision and language-model parameters use FP16 during pretraining to improve efficiency.
- B. Implementation details of Vision Langauge Model: The vision adapter takes concatenated class tokens from different vision-transformer stages as input and maps them through two linear layers, SiLU activation, and layer normalization.
C. Joint prediction of Articulated Objects
The articulated-object joint prediction module adapts GAMMA to improved multi-view point clouds and predicts object parts and joint parameters. DBSCAN then clusters inferred parts to support joint-parameter voting, with final performance reported on an unseen test set.
- C. Joint prediction of Articulated Objects: GAMMA is fine-tuned on improved multi-view point clouds because its single-depth-camera input creates a domain gap in this setting.
- C. Joint prediction of Articulated Objects: GAMMA extracts point features with PointNet++ and combines segmentation, offset, and joint-parameter estimation components.
- C. Joint prediction of Articulated Objects: The segmentation head classifies points as static, revolute, or prismatic, while the joint-parameter head predicts joint geometry including axis direction.
- C. Joint prediction of Articulated Objects: Training minimizes a combined loss containing segmentation, offset, and joint-parameter terms.
- C. Joint prediction of Articulated Objects: After training, frozen features are clustered with DBSCAN using predicted semantics and axis-aware shifts, and segmented parts vote for joint parameters.
- C. Joint prediction of Articulated Objects: Final performance is evaluated on an unseen testing set, with results presented in Table 5 for articulated-object joint prediction.
D.1. Grasp Pose Proposal Generation
RoboCodeX supports grasp-pose proposal through two complementary models and task-dependent physical preferences. Anygrasp generates collision-free parallel-jaw grasps from RGB and point-cloud inputs, while central lift produces top-down grasps around object centers.
- D.1. Grasp Pose Proposal Generation: Anygrasp uses a single RGB image and corresponding point cloud to propose collision-free grasps for parallel-jaw grippers.
- D.1. Grasp Pose Proposal Generation: The central lift method generates top-down grasp poses by focusing on objects’ central lifting points, which often align with their centers of mass.
- D.1. Grasp Pose Proposal Generation: RoboCodeX refines adaptive grasp selection with preferred position, approach direction, and plane normal parameters that vary by task and object properties.
D.2. Grasp Execution
Grasp execution combines a progressively decelerating pre-grasp trajectory with controlled gripper actuation and an API that exposes perception, manipulation, and motion-planning operations. Prompted preferences replace hard-coded grasp values so generated behavior can adapt to tasks and objects.
- D.2. Grasp Execution: The pre-grasp trajectory approaches the object from a predefined position while progressively reducing motion increments for precision and stability.The gradual deceleration is especially intended to protect lightweight or delicate objects from displacement or damage caused by abrupt movements.
- D.2. Grasp Execution: Closed-loop gripper actuation seeks a secure grip while limiting force to prevent damage during manipulation.
- E. Introduction of the APIs and the prompts: The Python API connects ROS interactions, perception utilities, object manipulation, path planning, trajectory execution, and explicit gripper control for generated robot programs.Prompted preferences specify gripper position, approach direction, and plane normal, enabling task-oriented alignments instead of fixed values.
F. Real world experiments
Real-world experiments evaluate RoboCodeX across Franka Emika Panda and UR5 robot arms without platform-specific task fine-tuning. The framework transfers between platforms by changing the robot configuration file, with reported task-specific successes on both arms.
- F. Real world experiments: The UR5 performed fruit and cosmetic manipulation tasks, while the Franka Emika Panda positioned a toy on a car.
- F. Real world experiments: Figure 10 reports real-world experiments on the Franka Emika Panda and UR5 robot arms.
- F. Real world experiments: Zero-shot transfer across Franka Emika Panda and UR5 robot arms required no platform-specific fine-tuning and only a modified robot configuration file.The experiments used different environments and object sets across the two robotic systems.
- F. Real world experiments: The code-generation setup provides ROS imports, perception functions, grasp-pose parsers, object attachment and detachment, gripper controls, pose movement, directional motion, joint-centered arc paths, and path following.The prompts also constrain workspace boundaries, discourage hard-coded values, and require generated output to contain only Python code and comments beginning with '#'.